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education and training, from control and accountability to performance and 
quality purposes. A classification model has been proposed in which many 
indicators can fit. It is based on two important dimensions of indicators: 

(1) the "message" relating to the information content, meaning, and 
signification of the indicator and (2) the "purpose" relating to the function 
and use of the indicator. Four main areas on the scale for the message 
dimension are as follows: exclusively descriptive, management and policy, 
performance, and quality. Three areas on the scale for the purpose dimension 
are indicators used for analysis, used for communication, and with a 
normative purpose. The two main driving forces for the use of indicators are 
public authorities and institutions. International and European achievements 
in the work on quality indicators are illustrated by the following: the 
Organization for Economic Cooperation and Development's INES (Indicators of 
National Education System) project, the Eurostat key data, and the European 
Training Foundation key indicators on vocational education and training 
(VET) . At the level of VET institutions, a wide range of performance and 
quality indicators have been developed across Europe in the Netherlands, 

Italy, Portugal, Denmark, and Western Australia. The advantages of using 

quality indicators can be achieved only if 10 conditions 5 design 

requirements and 5 requirements for processing and use- -are met. (Appendixes 
contain 77 references, design methodology for quality indicators, and 
glossary.) (YLB) 



***************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



CO 



s 



c 

| Indicators in perspective 

3 

O 

q The use of quality indicators 
1-1 in vocational education 
and training 

O 



LU 

O 

LU 

O 



European Centre for the Development of Vocational Training 



U S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
7 CENTER (ERIC) 

□/This document has been reproduced as 
/ received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



HAS BEEN GRANTED by 




t6 ^ educational RESOURCES 
information center (ERIC? 



T-« 



vO 



r- 

o 



^ERiC 




BEST COPY AVAILABLE 

2 




CEDEFOP Document 



Indicators in perspective 

The use of quality indicators 
in vocational education 
and training 

Authors: 

Wouter Van den Berghe 
Tilkon Consultancy 

November 1997 

First edition, Thessaloniki 1997 

Published by: 

CEDEFOP — European Centre for the Development 

of Vocational Training 

Marinou Antipa 12, GR-57001 Thessaloniki 



Postal address: 

PO Box 27 — Finikas, GR-55102 Thessaloniki 
Tel. (30-31)4901 11 
Fax (30-31)4901 02 
E-mail: info@cedefop.gr 
Internet: http://www.cedefop.gr 



The Centre was established by Regulation (EEC) No 337/75 
of the Council of the European Communities, last amended 
by Council Regulation (EC) No 251/95 of 6 February 1995 
and Council Regulation (EC) No 354/95 of 20 February 1995 




o 



A great deal of additional information on the European Union is available on the Internet. 
It can be accessed through the Europa server (http://europa.eu.int). 



Cataloguing data can be found at the end of this publication. 

Luxembourg: Office for Official Publications of the European Communities, 1998 
ISBN 92-828-2820-4 
© European Communities, 1998 

Reproduction is authorised provided the source is acknowledged. 



0 



Printed in Italy 



Foreword 



This report is one of the series of studies elaborated within the context of CEDEFOP's 
project on quality in vocational training. 

In conformity with the European Council Resolution of 5 December 1994 on the quality 
and attractiveness of the VET, CEDEFOP has commissioned and is currently commis- 
sioning studies on some key aspects of the debate on quality at European level. It should 
be clear that the Centre is working both on quality assurance and quality assessment 
issues. 

Quality being a multi-dimensional and relative concept, CEDEFOP has started its works 
on it with a synthesis of ideas and experiences in seven EU Member States comple- 
mented by additional information. This publication of "Quality issues and trends in voca- 
tional education and training in Europe" is actually available in English, German, Spa- 
nish and French. 

Concerning exclusively initial vocational education and quality, CEDEFOP has chosen to 
limit itself to a comparison of school-based quality concepts and practices in two coun- 
tries, which are well-advanced in this field: The Netherlands and Denmark. 

Two national reports, merged into one under the title "Quality Debate in Initial Vocational 
Education", have been drawn up. The report, published in English, gives an overview of 
the policy context on quality and a detailed presentation of the cases of ten schools in 
both countries, which have adopted different and often complementary approaches to 
quality assurance. 

Based on discussions with experts in the field, CEDEFOP realised that over the last de- 
cades the "quality wave" originally started within the manufacturing industry, has also hit 
the training institutions in Europe. 

Certification, and especially certification on the basis of the ISO 9000 standards being 
one of the most sought after certification mechanisms, CEDEFOP has published a study 
on the "Application of ISO 9000 Standards to Education and Training" in English and 
French. The study does not limit itself to presenting the ISO 9000 standards, but it also 
provides for an interpretation of each single paragraph in a European perspective. It also 
contains practical suggestions for their implementation and underlines the requirements 
set up before launching such an operation. Based mainly on the experience gained by 
their implementation in the United Kingdom, France, Germany, Belgium, The Nether- 
lands and Denmark, the report adopts a critical approach as to the real value added of 
this quality assurance mechanism in VET. 

In relation to quality assessment CEDEFOP has organised jointly with the Greek OAED 
and the German FHVR-Berlin a conference on "Approaches to the evaluation of Europe- 
an Training, Employment and Human Resource Programmes", aiming to contribute to the 
definition of priorities so that European Programmes can be more efficiently implemented 
and evaluation methods improved. 
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An international and interdisciplinary exchange of views and information on employment 
policy, vocational training and social policy aspects of the evaluation of European trai- 
ning, employment and human resource programmes took place during the two days of 
this conference in Athens. The interested reader may find the related papers in CE- 
DEFOP's panorama series in English, Greek, Spanish and German. 

Following this, CEDEFOP has proceeded to the analysis of the evaluation practices of 
quality aspects in vocational training programmes. Five countries have been studied and 
a synthesis report is in preparation. It will be available in English and French. 

Concerning both quality assurance and quality assessment, the Centre focused its atten- 
tion on the study of the quality indicators in VET, because the development of quality and 
performance indicators in vocational training and especially in continuing education, is 
still at a very early stage. 

Indicators being used in quite different ways, from control and accountability to perfor- 
mance and quality purposes, the need was felt to present the various types of indicators 
in the context in which they operate. Their presentation follows the author's own ap- 
proach and should be read according to his "message-purpose" model, introduced in the 
first chapter. 

As is known, indicators have been developed by a range of institutions at international 
and at national level. The work on quality indicators at international and European level 
forms the third chapter of the present study, where the OECD-INES project, the Eurostat 
key data or the European Training Foundation key indicators on VET, figure among other 
international achievements. 

Chapter four presents the work done by education and training providers either at regio- 
nal, local or even institutional level. 

Besides the critical presentation of the various activities on development and implemen- 
tation of quality indicators in VET, the present report provides for a classification of them 
as we have felt that this could contribute to a better understanding of their scope and par- 
ticular context, since there has been little co-ordination when they were designed. 

The main reason for the limited use of quality indicators in VET being the difficulty to meet 
the preconditions in the various stages of their elaboration, the study analyses the poten- 
tial problems and proposes ten criteria to be met when designing, processing and using 
them. 

CEDEFOP wishes to thank the author, Mr Wouter van den Berghe and all experts who 
have helped him in one way or another, for the present work, which we hope will shed 
light on the tricky question of quality indicators, their trends and use in the future. 



Tina Bertzeletou 
Project manager 



Stavros Stavrou 
Deputy director 
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Preface 



Following the Resolution of the European Council of 5 December 1994 on the quality 
and attractiveness of vocational education and training, CEDEFOP launched a series of 
initiatives in order to examine quality issues in vocational education and training, and to 
underpin the policy debate on these issues. This report on the use of quality indicators is 
a further contribution of CEDEFOP to this area. It complements earlier CEDEFOP reports 
on related quality topics, such as “ Quality issues and trends in vocational education and 
training in Europe!' and " Application of ISO 9000 standards to Education and Training. 
Interpretation and guidelines in a European perspective". 

The final chapter of the aforementioned CEDEFOP report “ Quality issues and trends in 
vocational education and training in Europe” included a number of recommendations for 
further research. One of these concerned the use of quality indicators in vocational edu- 
cation and training, as a means to assure and improve quality. The report listed a number 
of questions that deserved further examination: “What are criteria for determining quality 
indicators for vocational education and training? What types of indicators already exist, 
and which types are lacking? When and how can quality indicators be used? What 
makes quality indicators for VET different from those for general education? Should indi- 
cators be developed preferably at institutional or at macro-level?” 

This report can be considered as a first response to these questions. The initial objective 
of the study was to provide a better insight in the use of quality indicators as tools for 
quality implementation and assessment within vocational education and training (VET). I 
was asked to consider types of indicators that had been developed and/or are currently 
being used at European level, and examine their relevance as quality indicators. I was 
equally required to analyse the scope of the indicators examined, on what basis they 
were determined, and how they were used and implemented. Their value added was to 
be assessed, and appropriate classifications were to be developed. 

All of this was a challenging task in many respects. It had to be completed in a relatively 
short time span and with limited resources. The methodology adopted was based on the 
gathering and examination of a range of information sources (books, reports, articles, 
Internet, ...) as well as discussions and e-mail correspondence with experts at national 
and international level. 

I am much indebted to all the people who, now and in the past, have so willingly provided 
me with information and feedback on the complex issue of quality in education and 
training. They are too many to mention here - I hope the reading of this report may give 
them something in return. I appreciate enormously the financial support received from 
CEDEFOP, and the continuous encouragement from its staff member Tina Bertzeletou for 
undertaking this work. Finally I would like to thank the many education and training or- 
ganisations for whom I have been a trainer or consultant. These assignments always 
turned out to be enriching learning experiences that have shaped my ideas about what 
quality really means in an education and training environment. 



Wouter Van den Berghe, November 1997 
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Chapter 1 : Indicators 



1.1 What are indicators? 

1.1.1 Introduction 

This study seeks to examine the use and usefulness of quality indicators in education 
and training, with a focus on vocational training. Since “quality”, “indicators” and 
“(vocational) education and training (VET)” are fairly complex and multi-facetted con- 
cepts, the first parts of this report includes a number of sections expanding on these con- 
cepts and their interrelationships. This first chapter looks at the notion of indicators; the 
next chapters will discuss quality indicators and VET indicators respectively. 

Most people are familiar with ‘indicators', although they may not be used to the term itself. 
In fact, in everyday life, other terms and words may be used instead of indicators, e.g. 
‘rates’, ‘index/indices’ or just simply ‘figures’ or ‘averages'. Indicators are best known 
from the economic and financial worlds. Typical examples are annual inflation, unem- 
ployment rate, average staff costs, return on investment, GDP growth, trade balance, etc. 
These figures say us something about - hence “indicate” - the overall situation or per- 
formance of the economy, a market or a company. In fact most of these well known fig- 
ures can be considered as “performance” or “descriptive” indicators (terms which will be 
explained later). But we can also observe an increased use of more qualitative indicators 
about the economy and society such as consumer confidence, frequency of delays, life 
expectation, ... Moreover, the use of indicators has now penetrated most sectors of activ- 
ity, including the non-profit area. 

The development of indicators is not a recent phenomenon. Governments and private 
bodies with large and complex interests have always needed some form of 
“accountability" for the resources they provide. The current interest in indicators has 
precedents from at least the 18th century onwards. Indeed, the appearance and growth 
of public administration was accompanied by increasing demands for precise descrip- 
tions of public sector activity, and eventually for cost-benefit accounting. Already by the 
end of the 19th century in the developed world the gathering of statistics had become a 
large “industry” in the face of social, political and economic developments. 

The development of so-called “performance indicators” in North America and Europe 
was spurred in the first decades of the 20th century by the application of the theory of the 
so-called scientific management or “Taylorism”. This movement had achieved remark- 
able results in industrial settings. It focused on a detailed analysis of time and motion in 
occupations, and required efficiency measures and detailed cost-benefit accounting. In- 
terest awoke in other sectors, including education, to apply similar principles. These en- 
thusiastic efforts did not always yield the expected results, however. The “cult of effi- 
ciency” collapsed during the 1930s, but has revived occasionally since then. 

The development of economic indicators started seriously after the Second World War. 
Politicians, public administrators and business people increasingly needed up-to-date, 
synthesised, comparable and reliable data on a range of topics as inputs to policy and 
strategy development. A growing number of economic indicators (trade balance, indus- 
trial productivity, GDP per head, economic growth, etc.) were designed and refined over 
time. International organisations such as OECD, in cooperation with national statistical 
offices, played a major role in these developments. In a few decades, the economic indi- 
cators gained a solid position and status for international comparison. Similar to this 
movement at macro-economic level, many financial indicators (rates) were developed to 



measure the performance and potential of companies. Indicator sets have also been de- 
veloped for such areas as R&D, education, healthcare and public services. Such devel- 
opments are often of a more recent origin and appear to be less well coordinated. Often 
no mature and internationally accepted indicator sets have as yet emerged, but an initial 
framework may be available. 

Since the 1980s, the increasing focus on quality, performance and accountability in all 
sectors of activity has contributed to renewed interest in the use of indicators. A recent 
illustration of this trend is the success and increasing use of the ‘Balanced Score Card' 
[Kaplan & Norton, 1996] in different types of organisations. This instrument contains a set 
of indicators that relate to different perspectives of a company (financial perspective, cli- 
ent perspective, internal perspective and innovative perspective). The indicators are 
linked to company goals and critical success factors. 

The use of indicators in education and training is strongly related to the historical devel- 
opment of governments’ interest in the allocation and spending of their resources. In the 
early 20th century, it was also subject toTaylorist ideas. An anecdote: in the 1920s, some 
American school districts and universities found themselves calculating ratios between 
heating costs and results on Latin tests, all in the name of “efficiency”... More seriously, in 
the United States the first movements towards accreditation of institutions can already be 
observed from the 1920s onwards - a development which required the availability of 
objective information in an accessible format 

Thus, the need for objective information to support the monitoring of education systems 
dates back from long ago, and was fuelled by the gathering of education statistics. It is 
only relatively recently, however, that the concept of (internationally comparable) 
‘indicators’ has gained fertile ground. A first prudent attempt of the OECD in 1973 failed. 
The second attempt, in the early 1990s, was much more successful, however, and was 
complemented by the work of several other international organisations and associations 
(see Chapter 3). This also helped to revive interest at national levels. 

Measured by the amount of literature published, the interest in the use of indicators in 
education has rapidly increased since the 1980s, particularly concerning indicators for 
policy formulation (e.g. in relation to expenditure levels) and as regards performance in- 
dicators in higher education, [see e.g. Van Vught & Westerheijden, 1993]. This growing 
attention can be related to the governmental interest in these matters. A second and 
more recent drive for the use of education and training indicators has come from the 
quality movement, which is now increasingly penetrating this sector. Modern quality as- 
surance approaches require appropriate measurements, and often these take the form or 
lead to indicators. But since such measurements are often institution-specific, compara- 
bility may be problematic. 

The focus of indicators tends to shift over time - and this can also be witnessed in educa- 
tion and training. Initially, most attention is paid to input factors (not at least because this 
is often most easy to calculate). But after some time, interest grows to produce indicators 
on outcomes. In certain sectors - including education and training - the production of 
such outcome-based indicators is, however, much more difficult to achieve, for reasons of 
complexity, cost, intangibility and even lack of consensus about the value of particular 
outcomes. 



1.1.2 Characteristics of indicators 



So, what is an indicator? What is the difference - if any - with ordinary statistics? Let us 
make it clear immediately: there are no official or universally agreed definitions of indi- 
cators as they are currently used. In one extrerfle, indicators include all types of statistics 
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and statistical tables (and their graphical derivatives) about the measurable attributes of 
a system, an organisation, a person, a product or a service: averages, frequency counts, 
totals, medians, upper quartile, cross-tabulations, etc. In another extreme, an indicator is 
nothing more than a signal that a threshold has (or has not) been reached - possibly 
even a non-numerical threshold (e.g. whether something is available or not). 

In this report we stick to an interpretation of indicators as a subset of ordinary statistics - 
and thus by definition ‘numerical’. Like other statistically obtained figures, indicators are 
meant to offer ‘overall’ information about the state of a system, an organisation, a product, 
etc. Good indicators provide a maximum of broad and useful information with a minimum 
of figures. The word ‘useful’ is very important here. Unlike many other ‘neutral’ statistics, 
indicators always ought to be defined with a particular type of use in mind. The analogy 
with the dashboard of a car may be useful here. There are thousands of statistics which 
could be provided about the actual performance of a car, but the dashboard provides 
only those which are useful (or even essential) to the driver. 

A second distinction with other types of statistics is that indicators are meant to facilitate 
relevant comparison. Three types of comparisons should be mentioned. The first, and 
often most important purpose, is the changing value of the indicator over time (every 
month, quarter, year, etc.). The evolution of the fertility rate is a well-known example. In 
some cases, the absolute value of the indicator is of little interest compared to its timely 
evolution, which may become an indicator in itself (e.g. annual inflation), often expressed 
in percentages (e.g. 3.4% economic growth). 

A second possible purpose of comparison, is between different units (organisations, 
systems, products, ...), even if these are of different size and complexity (e.g. schools). 
Because of this comparison rationale, indicators are often expressed as an (arithmetic) 
fraction, in which the denominator includes a size factor (e.g. pass rates in schools are 
calculated by dividing the number of successful pupils by the total number of pupils). 

Another reason for comparison may be the need to measure performance against a 
standard (e.g. the number of qualified trainers), a goal (e.g. sales volume), or a reference 
value (e.g. an average assessment score) - we will use the term ‘standard’ for all of 
these situations. Again, the indicator may be expressed as a fraction (in particular if the 
standard is also expressed in this way, e.g. the fraction of graduates employed within 6 
months), as a percentage of the difference between the standard and the actual value 
(e.g. percentage of goals that have been achieved), or - in some cases - as absolute 
numbers (e.g. the number of courses cancelled). 

A typical characteristic of indicators - by some even considered as a criterion - is that 
they are constructed by using two or more variables of the system being considered (or a 
combination of a variable with reference values or standards). Thus, in this definition, the 
‘number of trainees’ is a statistical measure, while the ‘average number of trainees per 
course’ is an indicator. In some cases, more than two variables may be needed to create 
composite indicators (e.g. an overall quality or performance indicator; the total score on a 
composite scale). As said, however, there may be situations where a simple calculation 
based on just one variable may qualify as a useful indicator (e.g. the total number of 
complaints). Often, in such cases, relating such figures to a ‘size’ factor or ‘standard’ 
would be irrelevant or defy the purpose of the indicator. 

Some experts insist that all indicators should be quantifiable : real numbers that are to be 
interpreted according to the rules governing their formation. This certainly holds for most 
indicators; by defining indicators as a subset of statistics, in this report a similar logic is 
followed. But we recognise that there are cases were qualitative or graphical information 
may also serve as an indicator, in particular where it concerns complex systems. For in- 
Q*"nce, it may be difficult to define quantitative indicators for the organisation of a project, 
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yet it is possible to verify that it has achieved its goals or not. In general, such more 
qualitative indicators always include an element of judgement and hence subjectivity. 

This brings us to another area of debate: should indicators be restricted to what is objec- 
tively measurable? We do not think so, because in most cases full objectivity cannot be 
achieved. Banning subjective indicators would often make it impossible to get a full pic- 
ture of a complex environment. In a set of indicators it is therefore often legitimate and 
desirable to include subjective data, such as information on user satisfaction. One has 
often to accept that indicator sets include some degree of subjectivity, part of which can 
never be eliminated. Adequate use and interpretation thus requires that all stakeholders 
agree that these subjective indicators are no direct measures but only aggregated per- 
ceptions by individuals of a certain state (as well as of their interpretation of the questions 
which are asked about it). Even some apparently 100% objective indicators that are 
based on hard data may have been designed based on a subjective consideration of 
what is important and/or what should be included/excluded from the data. 

Anyway, ‘subjective’ indicators have to be accepted when ‘quality indicators’ are dis- 
cussed. As mentioned in Chapter 2, quality is indeed often a subjective (customer- 
related) and context-dependent concept. Almost by definition this excludes the use of 
‘objective’ indicators only. 



1.2 A first classification of indicators 

1.2.1 The “Message - Purpose” Model 

The examples already given illustrated that there are many different types, formats and 
possible usages of indicators. This will become even more obvious throughout the re- 
mainder of this report. There are thus no easy ways to categorise all the different types of 
indicators, not even when only a restricted area such as education and training is con- 
sidered. 

Nevertheless, we feel that it would be useful to propose a first classification model in 
which many indicators can fit, and which is useful for the further discussion in this report. 
The model is based on two important dimensions of indicators. The first dimension re- 
lates to the information content, meaning and signification of the indicator; we will call this 
the ‘message’. The second dimension relates to the function and use of the indicator; this 
will be called the 'purpose'. For both dimensions, the model includes a scale between 
two extremes. 

For the ‘message’ dimension, the two ‘extremes’ are purely descriptive / static / input- 
oriented on the one hand, and quality /dynamic / output-oriented on the other hand. We 
may distinguish four main areas on the scale: 

■ Exclusively descriptive indicators (e.g. distribution of students across study disci- 
plines); 

■ Management and policy indicators (e.g. trends in interest for particular course topics); 

■ Performance indicators (e.g. percentage of training courses that finishes on time); 

■ Quality indicators (e.g. the percentage of trainers with very good didactic perform- 
ance). 

In practice, of course, there are no sharp boundaries between these types of indicators; 
indeed, certain indicators may span a large area on the scale, depending on the context. 
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As regards the ‘purpose’ dimension, the two sides of the scale are measurement, analy- 
sis, assessment /evaluation on the one hand and normalisation /standardisation on the 
other. Here it may be useful to distinguish three areas: 

■ Indicators which are useful for measurement, analysis, assessment and evaluation 
(e.g. detect problems); 

■ Indicators that are mainly used for communicative purposes (e.g. point out trends); 

■ Indicators which have a normative purpose (e.g. for comparison with set goals or stan- 
dards). 

When we create a graph with those two dimensions, most indicators can easily be plotted 
as an area, as illustrated below. 
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Thus, most indicators occupy an ‘area’ in this model, rather than a specific point. It could 
be argued that an ‘ideal’ indicator should occupy a very large ‘area’ in this model, since 
this would imply that the indicator contains different messages and can be used for dif- 
ferent purposes. It should be noted that most indicators are concentrated alongside the 
diagonal from the lower left to the upper right corner. 

This model - which is not specifically tied to education and training - will be helpful as a 
reference frame for the indicators discussed further in this report. We stress, however, 
that it is a necessary simplification, and this mainly for three reasons: 

■ There are many other dimensions which could be considered, for instance the degree 
of objectiveness, the type of underlying calculations (%, fraction, ...), the subject con- 
sidered (system, organisation, item, attribute), etc. We will return on these briefly in 
Section 4.3. For the purpose of this report, these other dimensions seem less impor- 
tant, but this could be the case in other- contexts. 
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■ For both dimensions considered, we have defined a ‘scale’ going from one extreme to 
another. In reality, the evolution across indicators is often not so linear as suggested 
by the model. Alternatively, the categories could also be considered as subsets of 
each other. This will be illustrated by the discussion of the ‘message’ dimension here- 
after. 

■ Finally, the place of an indicator in an area of the model may vary depending on the 
context. For instance, a ‘pass rate’ may have a ‘performance’ or ‘quality’ character in 
one context, but a purely descriptive one in another. 



1.2.2 The message dimension 

As mentioned before, the ‘message’ dimension concerns the content, meaning and sig- 
nification of the indicator. At the bottom end of the scale, we are close to ‘ordinary statis- 
tics’: (exclusively) descriptive indicators merely summarise aspects the subjects being 
considered. In fact, such indicators may also be called 'key data’ or ‘key figures’. Some 
typical types of descriptive indicators are: 

■ Frequency distributions across different topics/areas. 

■ Resource allocations. 

■ Counts of particular categories. 

In general, descriptive indicators are relatively ‘static’ and ‘absolute’ — i.e. the message 
depends little on the context and time. Such indicators are also predominantly input- 
oriented (see Annex B for a short discussion of input and output characteristics) and use 
a limited number of variables (sometimes only one). 

The second area on the scale of the message dimension are the management and policy 
indicators. These provide information that can be used for defining policy and strategy. 
The figures may suggest that current policy is appropriate and/or indicate areas that 
should be included in further policy developments (e.g. trends in interest for particular 
course topics). Some examples of such types of indicators are: 

■ Trends and growth/decrease figures. 

■ Figures showing the relationship with a certain threshold (reached, exceeded, ...). 

■ Indicators on non-critical process factors. 

Obviously, the boundary between descriptive and policy indicators is not very sharp - 
and anyway depends on the context. Management and policy indicators are mostly 
based on a dedicated design that exceed simple data gathering. They yield a level of 
knowledge-construction that may anticipate or monitor decision-making. The problem- 
solving potential of a well-designed management or policy indicator can be very high. 

This brings us to the third area on the scale - and the one that is most discussed in the 
literature: performance indicators. Before we discuss their meaning, two important re- 
marks need to be made: 

■ There is no agreed definition of what a ‘performance’ indicator means. In fact, what is 
called a ‘performance’ indicator in certain publications, should preferably be classified 
as ‘descriptive’ or ‘management’ indicators. Even the OECD uses the term 
^performance indicators’ in a wide variety of situations. Thus, what are called 
‘performance indicators’ in this report may not correspond with the concepts discussed 
elsewhere. 
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■ The notion ‘performance’ seems to have a broader meaning in English than in its di- 
rect translations into other European languages. The very term “performance meas- 
urement” - which is at the basis of performance indicators - translates in many lan- 
guages into terms which have a somewhat negative connotation, especially when it 
concerns the performance of individuals. 

In this report we follow a more restrictive definition of performance indicators. They are 
figures that indicate the (degree of) performance for an important component of an entity. 
In most cases, performance indicators include a process or output element, which are 
often compared with input factors. For a racing car, the average speed achieved is an 
obvious example of a performance indicator. In education, the average period needed for 
students to obtain their degree, is also a performance indicator - at least when student 
performance is being considered (from the perspective of the institution it may be a man- 
agement indicator). 

Typical types of performance indicators are 

■ Figures about average speed, power, productivity, ... (mostly in industrial settings). 

■ Indicators in relation to delays, defects, problems, complaints, errors, ... 

■ Indicators on critical process factors. 

■ Indicators providing information on efficiency and effectiveness of operation. 

■ Indicators showing the degree of achievement of objectives, goals or plans. 

Performance indicators can be defined for different levels and functions within an organi- 
sation (management, design, operational level, customer interaction, ...). This is related 
to the fact that these indicators are in most cases linked to processes within organisations 
or systems; it is possible to define processes at different levels and for different functions 
within organisations. For instance, performance indicators could be designed to indicate 
the effectiveness of the recruitment policy (high level indicator), but also for the error-free 
completion of internal forms (low level indicator). A special category of performance indi- 
cators are those which are linked to the achievement of particular goals, objectives or 
plans. Some of these goals could be related to particular processes (e.g. x% of students 
are to pass their exams successfully), but that is not necessary the case (e.g. degree of 
absenteeism). 

Finally, we could describe quality indicators as particular performance indicators that re- 
fer to a quality characteristic or objective. We will return to this term in the next Chapter. It 
should already be stated that modern quality theory sees quality as a context and cus- 
tomer dependent characteristic; it follows that what are considered quality indicators in 
one context, are merely performance or even management indicators in another situa- 
tion. For instance, in vocational training, the relation between training and subsequent 
type employment is important, and indicators on that relationship can be considered as 
quality indicators. But that relationship is much less important in initial, general educa- 
tion, where the same indicator may be useful as a policy information only. 

Typical quality indicators are of the following types: 

■ Figures showing progress on the implementation of quality policy. 

■ Performance indicators in relation to quality goals and objectives. 

■ Figures related to the assessment of quality and performance as seen by the cus- 
tomer. 



So far we have listed the different “message” types of indicators (descriptive / policy / 
performance / quality) on a linear scale, with the recognition that there is some overlap 
between the areas. This categorisation is, of course, only approximate. For instance, 
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there are quality characteristics that rnay not be regarded as a ‘performance , at least in 
the strict definition we have given to the term (e.g. the quality of a material, the quality of a 
design). 

Instead of using a scale for the message type of indicators - a feature which was intro- 
duced to facilitate the construction of the “Message-Purpose” model - one may also see 
these as subsets of each other. Indeed, most quality indicators have performance char- 
acteristics, performance indicators are useful for policy analysis, and management indi- 
cators have of course a descriptive function as well. This can be displayed graphically as 
follows: 




Statistics. 






1.2.3 The purpose dimension 



The second dimension of the “Message-Purpose” model concerns the function and use 
of the indicator. For the purpose dimension, we have retained three areas. 

The first purpose of indicators concerns their direct measurement function. Measurement 
is undertaken to undertake some ana/ys/s, assessment, monitoring or evaluation. A typi- 
cal example in a training context is the didactic score obtained by a trainer (obtained 
through a survey across trainees). Such indicators try to summarise the key outcomes of 
an often much more complex evaluation or measurement process. Typical types of indi- 
cators for assessment purposes are constructed on the basis of: 

■ Outcomes of survey results which are quantifiable or can be codified; 

■ Automatic measurements (tangible products); 

■ Frequency counts and cross-tabulations of errors, problems, deficiencies, ... 




In general such indicators can only be interpreted within a specific evaluation context. 
For instance, the number of errors made per trainee only makes sense if you understand 
the type of training and the profile of the trainees. 




The second area are indicators for communication purposes. These indicators are typi- 
cally designed and used for informing relevant stakeholders or the outside world of the 
state of development of a particular system, organisation, process, etc. They may also be 
used for archiving and documentation purposes, in order to be able to detect trends over 
a long period. In the field of education and training, most ‘indicators’ published at na- 
tional or international level (see Chapter 3) can be considered to fall in this category. We 
recall also our previously used analogy of the dashboard of a car, where the different 
types of lights and meters are displayed to inform the driver. More general, in a quality 
management context, such 'communication purpose’ indicators serve to raise aware- 
ness, promote a culture of fact and figures, and may facilitate discussion between differ- 
ent parties (by making complex facts visible in a simplified way). 

Typical applications of such indicators are: 

■ An input to discussions. 

■ Publicity and external communication. 

■ A basis for the establishment of action plans. 

■ Comparisons with the past. 

■ Pointing out new developments and trends. 

Because of such utilisation, these indicators need to be easily understood and inter- 
preted by all those concerned. 

An even more specialised function of indicators is when they are used for normative or 
standardisation purposes - the other end of the ‘purpose’ scale. Such indicators are 
used to show to what extent the actual situation deviates from an established norm, pre- 
defined standards, set goals, etc. For instance, imagine a norm that 80% of courses 
should be considered at least ‘good’ by the trainers; the indicator itself may show that the 
actual result is 84%. As this example illustrates, the ‘norms’ may themselves be ex- 
pressed in the form of an indicator. Indicators that are linked to strategic goals and op- 
erational objectives are mostly normative in nature. Other indicators of the same category 
are those that include comparisons with similar entities. 

It ought to be clear that these purposes may overlap. In fact, just like for the ‘message’ 
dimension, one may not only consider the three areas as distinct groups on a scale, but 
also as a series of subsets. Nevertheless, it may not be desirable to have just one indi- 
cator that intends to serve different purposes and is used in different contexts. Since such 
an indicator is necessarily a compromise, none of the different forms of utilisation may be 
entirely appropriate. 



1.3 Rationale for the use of indicators 

1.3.1 Introduction 

In the beginning of this report we mentioned already that there are two main driving 
forces for the use of indicators. On the one hand, there is a need for the owners and dis- 
tributors of massive resources (such as in education) to obtain a viable picture of the 
system. On the other hand, there is an internal drive emerging from within organisations 
that see the need for more modern and professional management approaches - hence 
including measurement. The first drive is the oldest and still the most important one. In 
both situations, stakeholders will seek to define indicators that reflect as exactly as possi- 
ble an aspect of the state of a system. 
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Over the last two decades the use of indicators has tended to increase in most sectors of 
activity, including education and training. This is not the place to discuss the origin of 
these movements; let it just be mentioned that this development results from a number of 
mutually reinforcing trends: 

■ the accountability and ‘new public management’ movements, calling for greater clarity 
and more objective measures of performance and control; 

■ the adoption of quality assurance and quality management approaches in organisa- 
tions - and, more general, more professional management methods; 

■ the internationalisation of economy and society, creating a need for comparable data 
across borders for entities of quite different scope; 

■ the advances in information and communication technology, which have greatly fa- 
cilitated the collection, gathering and comparison of information. 

Such developments reinforce the need for ‘hard’, quantifiable data in an easily accessi- 
ble and interpretable form. Hence, indicators emerge quite quickly. 

Using indicators has also its side effects, and one of these is the temptation for perfec- 
tionism. In an ideal situation, it may be argued, a limited set of ‘perfect’ indicators might 
cover the whole spectrum. In practice this can never be achieved. Fortunately, this is not 
necessary nor desirable. Indeed, indicators are only a tool, not an end in themselves. It is 
an illusion to think that a perfect set of indicators can ever be constructed and act as a 
substitute for interpretation. Moreover, every effort spent on creating and developing indi- 
cators should always be balanced by the use that will actually be made. 



1.3.2 Use of education and training indicators by public authorities 

The use of indicators in education and training systems varies considerably across 
countries and even regions. It would have led us to far to make an inventory and com- 
parison of the current state of the art (this is, however, done for international organisa- 
tions in Chapter 3). When we consider ‘public authorities’ or ‘governments’ in the sequel, 
we generalise to a large extent. It should be obvious that most of the statements made 
will apply in very different degrees to different countries. Indeed, there are countries 
where education and training indicators hardly exist, whilst other ones have already a 
certain tradition in this regard. 

As we will see in Chapter 3, there is currently a considerable investment of international 
bodies, supported by national administrations, in the development of “indicators" in the 
field of education and training. This reflects a growing belief among governments that the 
existence of such indicators might stimulate informed argument on desirable develop- 
ments, e.g. in relation to curriculum development, teaching practice, services provided, 
fundinq levels, etc. This is the ‘communication’ purpose of certain indicators (see Sec- 
tion 1.2.3). 

Similarly, there has often been a desire to introduce indicators as a means to improve 
quality, in particular the improvement of educational and/or administrative (through the 
effect of highlighting particular strengths and weaknesses of institutions). Linked to this 
has often been the wish to improve the accountability of the institutions supported, by 
indicating in an “easy to communicate” way their quality and performance. In some 
cases, performance and quality indicators were intended to become the basis of an 
emerging quality control system (supported by other control mechanisms). Overall these 
efforts do not seem to have been successful, often because of disputes about the rele- 
vance of the indicators proposed. 



One - sometimes hidden - reason for introducing quality or performance indicators has 
been to encourage competition among education and training organisations (or within 
them). The underlying assumption is that competition produces improved educational 
practice, encourages quality performance of individuals, improves efficiency, etc. Not 
surprisingly, such views are not very popular with education and training staff. In our 
opinion, there is increasing evidence that stimulating fair competition amongst VET insti- 
tutions indeed leads to higher quality. The availability of system wide quality indicators is 
likely to facilitate such competition, because it leads to increased transparency. 

However, the most important driving force for introducing and analysing education and 
training indicators has often been the increasing concern of public authorities for better, 
more effective and efficient spending of resources (the 'assessment' and 'normative' pur- 
poses). In particular since the 1980s, it would appear that this resource-focused concern 
has been the primary force behind the development of indicators. This statement can be 
illustrated by the fact that there is much more interest and effort going on in relation to 
performance indicators than on quality indicators (our definitions). In Europe and par- 
ticularly North America, performance indicators have often been used to decide how and 
when to cut resources, and sometimes also to force changes in curriculum and allocation 
of education resources. It is interesting to note that in the 1993 reform of the Swedish 
higher education system, it had been envisaged that 5% of resources allocated would 
depend on a number of quality indicators; however, this has not yet (Summer 1997) been 
implemented and seems to be abandoned as an idea [Bauer & Kogan, 1997], 

Another important reason - originally the most important one - for governments to de- 
velop indicators has been to produce a valid, reliable and complete picture of the per- 
formance and quality of the education and training system at different levels (the 
'descriptive' and ‘management’ message). A picture that could be compared with the 
past situation and that could serve as a reference frame for comparison. And finally a 
picture that has a minimum of components, so that it can easily be understood and inter- 
preted. Meeting this need can only be achieved through a combination of a more holistic 
and qualitative approach (based on reviews, audits, surveys, ...) combined with evidence 
provided by data, from which indicators can be constructed. 

Other governmental objectives and considerations have been to make the financial and 
other reporting consistent across all institutions of a similar type - and even comparable 
with other types of organisations. 



1.3.3 Introduction of indicators at institutional level 

Although public administrations and governments are the driving force behind the intro- 
duction of indicators in education and training, there is a growing upstream movement 
from institutions as well. This is linked to the increasing recognition of the applicability of 
professional management methods in education and training institutions. Such man- 
agement approaches rely much more on facts and figures: for monitoring the current 
situation, for defining policy and operational objectives, and for identifying important 
trends. Well-designed indicators can be very instrumental in this regard. For instance, a 
specialised training institute seeking to increase the number of handicapped trainees 
could develop and use indicators to examine whether it was successful in this regard. Or 
if the purpose would be the reduction of administrative expenditure, an indicator on the 
proportion of resources spent on administration is obviously useful. 

It is true that in service environments (including education and training) it is more difficult 
than in industry to gather data, undertake automatic measurements and monitor process 
development. We have found, however, that there is often a surprising volume of infor- 
Tf * : on already available which, with some manipulation, can be turned into useful indi- 
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cators. For instance, few institutions measure the efficiency of the activities undertaken. 
However, many documents are registered and/or date stamped, which would make it 
possible to measure at least duration and delays. Together with an estimate of man- 
power input, a first crude indicator of efficiency might fairly easily be constructed. 

The first challenge for an institution interested in developing a useful indicator set, is not 
so much the definition of new information gathering mechanisms and measurements 
processes. Rather, it will need to improve standardisation, registration, and access of 
critical data collection, and possibly redefine its frequency. The extent of accuracy will 
depend on the purpose and envisaged use of the indicator. 

It should be kept in mind that an institution’s development can be described with indica- 
tors only to a certain extent. Indeed, many education and training institutes (in particular 
in general education) have a complex series of objectives that cannot easily be captured 
by a few quality or performance indicators. For instance, how to design an indicator on 
the ‘cultural’ or ‘social’ added value generated by a school? 
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Chapter 2: Quality and quality indicators 



2.1 Introduction: the Quality movement in education 

The concept of “quality indicators” cannot be fully understood without comprehension of 
the quality concept itself, in particular its modern significance. This first section highlights 
some points in relation to the origin and rationale of the quality movement, while the next 
one will deal with the concept of quality itself. The final section of this chapter then devel- 
ops the notion of a “quality indicator”. 

Since the 1980s the topics Quality, Quality Assurance and (Total) Quality Management 
have become a central preoccupation of hundreds of thousands of organisations in 
Europe. The industrial world was first to recognise that much of the Japanese economic 
success could be attributed to the consistent focus on quality assurance and quality im- 
provement. It is already more than a decade since European firms have started to adopt 
the concepts and methods of Total Quality Management - in short 'TQM' - to meet the 
quality level which customers expect and for improving continuously the quality of prod- 
ucts and services they deliver. Similar quality movements started to penetrate other ar- 
eas, including service sectors. In particular since the early 90s - this varies somewhat by 
country - there is increased and renewed attention for quality issues in education and 
training. 

It is not the purpose of this report to discuss the different factors which have contributed to 
the increasing emphasis on quality issues in the education and training world - and how 
governments, institutions and learners are coping with these challenges. These topics 
are dealt with in more detail in the recent CEDEFOP report "Quality Issues and Trends in 
VET in Europe ’’[Van den Berghe, 1996]. Therefore, only some general points are re- 
called in the next paragraphs. 

The first observation is that quality is not a new subject in education and training. Institu- 
tions, teachers, administrators, policy makers and learners have always been concerned 
with quality. Indeed, the quality of an education or training provider eventually depends 
on the performance of the learners. Even without a formalised ‘quality’ approach such as 
TQM or ISO 9000 [Van den Berghe, 1997], schools and training providers have needed 
methods, norms, procedures and standards to ensure the quality of their provision. But it 
is equally true that, traditionally, quality has often been interpreted fairly narrowly, fo- 
cussing on particular features of the education and training services delivered. 

This brings us to the second general observation: in education and training, just like in 
most other sectors, the dominant quality ethos tends to change over time. The current 
trend is one whereby quality concerns are increasingly focusing on the total effectiveness 
of an education and training provider, whether that is a vocational school, a university or 
a private training institution. This tendency mirrors the industrial developments where 
quality considerations are moving towards the organisational capacity to deliver high 
quality goods and services (the systems thinking behind the Total Quality Management 
concepts). 

A third important observation is that the current concerns for quality in education are no 
isolated or temporary phenomenon, but are part of broader macro-economic trends. It 
does not imply that 'older' types of quality considerations (e.g. didactics) are no longer 
valuable. Rather, ah effective 'Total' Quality strategy in education should be able to incor- 
porate the more traditional quality perspectives, 
o 
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A fourth general observation is that the ‘new’ approaches to quality assurance and man- 
agement in education and training mean something different for each type of education 
and training. Simply stated: 

■ a shift in quality focus for general and vocational schools, from emphasis on the quality 
of the teacher towards the performance of the institution as a whole; 

■ the introduction of new or additional quality control mechanisms in higher education; 

■ the creation of quality assurance systems and performance related mechanisms in 
continuing education and training. 

A common characteristic for all types of education and training is the increased concern 
for the performance of the learner (the effectiveness of learning) and the effectiveness 
and efficiency of the provider as a whole. This explains also the interest in TQM and 
quality assurance approaches. 

A final observation: notwithstanding the increased emphasis on organisational factors as 
a prerequisite for quality assurance, it is recalled that no effective learning can take place 
without high commitment of the teacher or trainer. The corollary is that quality ap- 
proaches - including TQM and ISO 9000 - are doomed to fail if they do not support the 
inner motivations of teachers and are able to sustain or increase their commitment. 



2.2 Quality in Training 

2.2.1 Quality concepts and terminology 

Quality is now considered as a multi-dimensional, relative (customer dependent) and 
context-related concept. Therefore, it is not possible to give a unique definition that fits all 
circumstances. It is possible, however, to distinguish some important viewpoints and per- 
ceptions to quality: 

■ quality as excellence, as something special; 

■ quality can a measure of characteristics (product-oriented quality); 

■ quality as conformance to specifications (process-oriented quality); 

■ quality as the fulfilment of customer expectations; 

■ quality as optimal benefit/cost ratio (the value approach). 

Such quality perceptions may apply, alone or in combination, to any type of product or 
service, including vocational education and training. But different people may hold differ- 
ent views about the same object. In the modern quality movement, the three last view- 
points on quality which predominate: conformance to specifications, customer orienta- 
tion, and optimal value. They may be summarised in the slogan “fit for purpose". 

In order to achieve quality in a particular outcome, the quality of the preceding processes 
needs to be ensured. Two important concepts in this regard are quality of design and 
quality of conformance. Quality of design is the capacity of an organisation to design, 
plan and specify products and services that will be considered to be of quality by the end- 
users. Quality of conformance is the ability of an organisation to meet the design, plan- 
ning and specifications, and to respect agreements. An organisation has to be strong in 
both areas if it wants to produce quality outputs all the time. In general this requires a 
systematic quality approach based on the principles of TQM. 

The concepts “quality of design" and “quality of conformance” are fully applicable to the 
provision of VET. Quality of design refers to training specifications that meet the require- 
ments of the particular profession, skill or job ( Are the programmes suitable and 



relevant?). Clearly, the involvement of social partners and/or the customers will facilitate 
this task. Quality of conformance is the capacity of a training provider to deliver the pro- 
gramme and meet the specifications (Is the provider performing well?). It is at this level 
that quality assurance and quality control mechanisms play an important role. 



2.2.2 Quality perspectives in education and training 

Before the emergence of ‘modern’ quality approaches and concepts, the education and 
training world had already developed its own quality tradition. The different viewpoints 
from which quality in education and training has been considered can be summarised as 
follows [see e.g. Van den Berghe, 1995]: 

■ Quality from a didactic and/or pedagogical point of view (education quality seen as the 
optimisation of the teaching and learning process). 

■ Quality from a (macro)-economic point of view (education quality seen as the optimi- 
sation of the education and training costs). 

■ Quality from a social or sociological point of view (education quality seen as the opti- 
misation of the response to social demand for education). 

■ Quality from a customer point of view (education quality seen as the optimisation of the 
demand). 

■ Quality from a management point of view (education quality seen as the optimisation 
of the organisation and processes of education). 

The order in which these different viewpoints are listed reflects to some extent the shifts 
in emphasis over the last decades in many European countries. However, it is not so 
much a question of replacing “old” paradigms, but rather one of adding “new” dimensions 
- which reflects the growing complexity of the education system and the objectives it has 
to meet. 

Although the term “quality” is gradually finding its way into the VET legislation of Euro- 
pean countries, the notion is not necessarily well explained - leaving some ambiguity as 
to which perception of quality has been in mind of legislators. Some formulations seem to 
suggest that quality is considered as an absolute, measurable concept. But recent laws 
in some countries support more the view that quality in education and training must be 
related to the values and aims and objectives of three user groups: the students, the la- 
bour market purchasers, and the society in general. Since these groups may have differ- 
ent expectations and needs, political objectives eventually tend to become the mea- 
suring rod for quality. Thus, attempts to define quality precisely in education and training 
lead to political discussions and choices. 

Thus, while an exact quality definition - which essentially comes down to specifying the 
criteria for the design of programmes (“quality of design”) - is somewhat problematic in a 
VET environment, there are less problems with the notion of quality assurance : how to 
ensure that the quality specifications can be met. This process logic that is less subject to 
political debate, but has more to do with identifying the factors and operational charac- 
teristics which are most effective. 
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2.3 Defining quality indicators 

As is obvious from the previous discussion, the meaning and value of "quality indicator s" 
will depend strongly on the context (a phenomenon which applies to other areas than 
education and training as well). In Chapter 1 we described quality indicators already as 
“ performance indicators that refer to a quality characteristic or objective". A more precise 
definition is proposed below. 



“A quality indicator is a figure which is helpful for the assessment of a 
quality characteristic or the achievement of quality objectives ” 



Let us briefly comment on this definition. The first thing to notice is that ‘quality’ itself is not 
defined. The different perspectives - and possibly disagreements - on quality also apply 
to ‘quality indicators’. What is a quality indicator for one person, may be an ‘ordinary’ in- 
dicator for another. It must be said, however, that in VET the debate is not so much about 
whether something impacts on quality, but rather its relative importance. For instance, 
nobody would dispute that attitudes developed by pupils is a quality characteristic of their 
competence development, but teachers, employers and pupils themselves are likely to 
disagree about the relative importance of this (for instance in relation to knowledge ac- 
quisition). 

The underlined words in the definition also require some comments: 

■ “figure”: this relates to the view, taken in this report, that an indicator should be nu- 
meric; 

■ " assessment ’: this reflects the fact that quality is not seen as something absolute, but 
depends on the views of the user; 

■ “characteristic”: this refers to a product or the output of a service; in general it reflects 
the “quality of design” performance (e.g. “How adequate is a programme for a particu- 
lar feature?')] 

■ “objectives": this concerns standards (internal or external) or set objectives which have 
to do with quality aspects; often these reflect the degree of “quality of conformance” 
(e.g. 'Was the programme delivered on-time?’). 

In the first chapter we have already shown the link between quality indicators and other 
types of indicators: performance indicators, management/policy indicators and descrip- 
tive indicators. It seems also useful to relate quality indicators to some other quality con- 
cepts, such as quality standards or objectives. These relationships are symbolically dis- 
played in the graph on the next page, which shows that: 

■ quality indicators are only a small part of an overall quality system; 

■ quality indicators are only one of many representations of quality characteristics; 

■ some quality standards, norms or criteria can be represented as quality indicators 
themselves; 

■ some quality goals and objectives can be expressed in the form of particular values of 
quality indicators. 
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Overall, the graph indicates that quality indicators are no substitute for a quality system or 
for quality standards; they are a tool that can help in assessing - and hence improving - 
quality. 




When considering the “purpose" dimension of our “Message-Purpose" model, we are 
tempted to conclude that quality indicators in education and training have already been 
introduced for each of the three main purposes, but in particular for communication and 
assessment purposes. Some attempts to use quality indicators in a normative context 
have been undertaken, for instance in higher education, but these have met much resis- 
tance. This can be related to the fact that (1) the use of quality indicators in education and 
training is still in its early stages, and (2) there is often lack of consensus about the es- 
sential quality characteristics, as well the possibility to measure these and represent 
them in a numerical format. 

In conclusion, quality indicators are - or ought to be - carefully designed measures that 
provide reliable information about particular quality characteristics of a system or about 
the attainment of certain quality objectives. The value and usefulness of particular quality 
indicators can only be assessed if there is clarity about the underlying quality concept. 
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Chapter 3: International VET-related indicators 



3.1 Introduction 

This chapters considers indicators - quality indicators and others - which have been de- 
veloped for Vocational Education and Training (VET) at European level. Please note that, 
despite the long tradition of gathering statistics on education and training at European 
level by organisations such as Eurostat, OECD and UNESCO, the emergence of interna- 
tional indicators is a relatively recent development. This will be obvious from this chapter. 

Not unexpectedly, another important finding from our research was that most interna- 
tional education and training indicators concern the general education system. Some- 
times vocational education is covered, but not necessarily in a transparent way. Far less 
attention has gone to vocational education and training, both at initial and continuing 
level. One important reason seems to be that information on VET is more difficult to col- 
lect and process than on general education. This holds in particular for continuing voca- 
tional training (CVT). Other reasons are the considerable differences between European 
countries in the organisation and characteristics of the VET system. This makes it very 
difficult to develop meaningful internationally comparable indicators. A third reason for 
the limited number of international VET indicators is that, in many countries, the VET 
system is undergoing rapid change; this would render time series of indicator values 
quickly obsolete. 

Detailed information about these differences and developments can be found in many 
other CEDEFOP documents, and there is no point in repeating these here. We would 
like, however, to draw attention to the distinction between “initial” and “continuing" voca- 
tional education and training. In this report, we define “initial vocational education and 
training" (IVET) as the vocational part of the secondary education system, i.e. leading to 
recognised qualifications at that level. In Europe, we have two main types of such sys- 
tems. One is mainly or exclusively school-based, with teaching predominantly being 
given in technical or vocational schools, which are firmly embedded in the traditions and 
organisational forms of the overall national school system. The second type is predomi- 
nantly apprenticeship based, with a large part of the training taking place within, and un- 
der the responsibility, of firms. 

By contrast, "continuing vocational education and training" (CVET) consists of vocational 
training programmes and courses for people who are (or could be) at work and/or have 
already some kind of general or vocational qualification (in some countries, such provi- 
sion is called "adult education”). Although part of such training could lead to recognised 
qualifications, continuing education and training displays many characteristics which 
make it different from IVET, such as: 

■ the existence of a very broad range of providers of different types, both public and pri- 
vate; 

■ programmes are mostly measured in terms of days and weeks, rather than months and 
years; 

■ training that is often targeted at specific groups: unemployed people, workers from 
certain sectors; 

■ the existence of several forms of state intervention or subsidy. 
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In reality, the boundary line between initial and continuing vocational education and 
training is not clear, in particular when: 

(1) CVET is delivered by institutions also providing secondary education level qualifica- 
tions, 

(2) when it leads to qualifications at secondary rather than advanced level, and/or 

(3) when it is fully subsidised by the state, or when its delivered immediately after a first 
qualification. 

Although a discussion of national education and training indicator sets is beyond the 
scope of this report, we considered it useful to give a French and a UK example. In the 
annual French publication about the state of the education and training system, L'etat de 
I'Ecole, the importance of ‘results’ and ‘outputs' - and hence quality - has gradually in- 
creased and been refined over the years. The most recent edition includes 30 indicators, 
of which several could be considered as quality or performance indicators [Ministere de 
I’Education, 1996]: 



9 Relationship between qualifications and salary 
1 0 Relationship between qualification level and social position 
1 3 Performance in French and mathematics of pupils entering the sixieme 

1 9 Differences in lycee results (succes in baccalaureat) 

20 Relationship between secondary schoool qualifications and career patterns 
26 Relationship between higher education qualifications and career patterns 

29 Continuing education activities 

30 Professional advancement of employees having received continuing 
education 



Please note that the 19the indicator on differences success rates follows a process logic: 
it takes into account input factors (i.e. age and social background of students) in order to 
measure the quality of these schools. The term actually used is the ‘added value’. 

A typical British phenomenon is the appearance of 'league tables' about all kinds of edu- 
cation establishments (including universities). Such league tables have appeared in the 
press (e.g. the Times, the Guardian, the Times Higher Education Supplement, ...), and 
may also be available on the Internet. The league tables are in general based on indi- 
cators constructed from publicly available data sources. As an example, we give some 
indicators used by the Guardian for classifying certain types of secondary schools [The 
Guardian, 1997]: 



1 . % of 1 5-year-olds getting 5 or more grades A-C 

2. % of 15-year-olds getting 5 or more grades A-C ranked by improvement over 
last year 

3. Average point score per A/AS level entry 
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If the success rate of pupils on certain examinations is considered as a quality outcome, 
such indicators are definitely quality indicators. It goes without saying that such indicators 
may only be useful for comparison across schools, if grades are delivered in a similar 
and standardised way (which is often not the case in European VET institutes). 

These two small examples already illustrate (1) the possibly very different approaches 
(both in message and purpose) which education and training indicators may serve, and 
(2) the fact that at national level, it is more easy to define relevant and specific quality in- 
dicators than at international level. In the remainder of this report we will only look at in- 
ternational VET indicators. We will start with the outcomes of the OECD INES project, 
because it is the one best known, and no doubt the one initiative that has so far had the 
most impact. 



3.2 The INES project of the OECD for education indicators 

The OECD was founded to promote economic growth, employment, and a rising stan- 
dard of living in the Member countries. It has developed a reputation for its in-depth 
analyses in all sectors of economic importance, including education and training. A par- 
ticular feature of many OECD studies is their emphasis on data and figure. To that end, 
the OECD has developed over the years a number of very important databases with in- 
ternational statistics. In many economic areas, the ‘OECD indicators’ are well known and 
are often considered as a standard. 

Since OECD countries spend about 12% of total public resources on education, the 
overall economic importance of education is not in doubt. Already more than a decade 
ago it had become clear that gathering, classifying and publishing education statistics 
was not enough. Like in other working areas of OECD, it was felt that the development of 
a limited set of international education indicators would help to strengthen the knowledge 
about the performance of education systems - and hence be instrumental in the imple- 
mentation of effective education policies. These considerations led to the launch of the 
INES project. The overriding goal was to develop, in a number of successive phases, a 
limited set of comparable indicators of national education system (including training ele- 
ments where available). Outputs were to be achieved through a collaborative framework 
consisting of a Technical Group and specialised networks responsible for the develop- 
ment of indicators in specific areas. 

The INES project began in 1988 with a feasibility study. By 1991 a first, preliminary set of 
indicators was developed, and published in 1992. Although this first set was not ideal, it 
signified a distinct political will in the OECD area to build up a comparative knowledge 
base for education. The publication of "Education at a glance" in 1992 attracted a lot of 
attention from the mass media, causing pressure to produce a second, more up-to-date 
and reliable edition, with increased comparability of data. 

Since 1992, three further versions have been produced [OECD, 1996], and the fifth one 
was in preparation when this report was finalised. Throughout the different stages, the 
reliability, comparability and political relevance of the indicator sets were systematically 
enhanced, and many new indicators were developed. Data collection was improved and 
speeded up, in collaboration with EUROSTAT and UNESCO. Whilst the second edition 
was essentially an improved and more up-to-date version of the first edition, the third edi- 
tion had several new indicators, as well as a larger coverage of countries. 

The fourth edition (1996) also included several new indicators. However, in order to keep 
the publication manageable, some existing indicators were dropped, in particular those 
which were not expected to change a lot from one year to another. An important new de- 



velopment was that out of the 43 indicators, almost one third concerned outputs and out- 
comes (in the previous editions, this proportion had been much lower). These 
‘performance’ or ‘quality’ indicators are listed below: 



R6 Student achievement in mathematics and science 

R7 Student differences in mathematics and science achievement 

R9 Difference in achievement between 7th and 8th grade in mathematics and 
science 

RIO Gender differences in mathematics and science 

R30 Literacy and the adult population 

R31 Adult literacy by level of educational attainment 

R32 Literacy skills of younger versus older persons 

R33 Adult literacy by gender 

R21 A Unemployment by level of education 

R22B Youth unemployment by level of education 

R22 Education and earnings from employment 

R24 Unemployment rates of persons leaving education 



It should be noted that the ‘achievement’ and ‘literacy’ indicators were derived from the 
IALS and TIMSS surveys - see Sections 3.5.2 and 3.5.3. below. 

The 1996 edition of 'Education at a Glance' was an OECD best-seller, and the interest for 
the publication and its further editions remains high. The INES-project currently involves 
the continuous cooperation of education authorities, public servants, researchers and 
statisticians in 30 countries. Considerable parts of the publication and the underlying sta- 
tistics are available freely from the OECD web site. 

The fifth edition - under preparation when this report was being finalised - would have 
the following features (based on indications provided): 

■ More attention to the analysis and discussion of the indicators. 

■ A relatively stable number of indicators, based on an overall policy to (approximately) 
replace one third, to keep one third and to modify/redefine one third of the indicators. 

■ The new indicators would cover areas such as the learning environment, teacher time, 
equity of opportunities, the cost per degree delivered, and continuing education and 
training (CET). 

The CET indicators would cover: 



1. Patterns of participation in CET (participation frequency, number of courses, duration). 

2. Demand characteristics of CET (by age, genders, level of education, labour market 
status). 

3. Supply characteristics of CET (providers, media of instruction, financial support, 
barriers). 



ERIC 



22 



31 



However, since most of the data sources are the IALS survey, the number of countries 
concerned is very limited. 

Although the attention to quality aspects has increased over the different versions, the 
number of 'real' quality indicators is still very modest. The development of finer and more 
specialised quality indicators would theoretically be possible, but would face two main 
problems. First, we think it would be almost impossible to get international consensus 
about specific quality characteristics, and/or the relative importance of certain quality as- 
pects. Secondly, the cost and resources required to collect and process data on quality 
issues would be very considerable. 

Some of the lessons already learned from this ongoing project are that good indicators 
improve the quality of the debate. There is strong interest in good, undisputed indicators 
-‘Education at a Glance’ has even been part of the Parliamentary debate in some coun- 
tries. There are already some indications of change that are attributable to the publica- 
tion, and further research has been stimulated. National adaptations (e.g. in Flanders, 
see De Groof e.a., 1995) have already appeared. Last but not least, the success of this 
project and the acceptance of the validity of the indicators seems to be directly linked to 
the shared ownership of the Member States. 



3.3 The European Union 

3.3.1 Introduction 

Like the OECD, the European Union has a long tradition of gathering, processing and 
analysing data in a wide range of areas, both in quantitative and qualitative form. The key 
actors in this process have been EUROSTAT, the Luxembourg based statistical office, in 
cooperation with the different Directorate Generals (DGs) of the European Commission 
(EC) in Brussels. In the field of education and training, the most relevant department is 
DG XXII (responsible for education, training and youth), but some activities in relation to 
education and training are also managed by other DGs, in particular DG V (training as 
part of the Social Fund activities). Some data collection - and even more data analysis - 
is undertaken in EC related bodies such as Eurydice (see Section 3.3.2 below), the 
European Training Foundation (see Section 3.3.4), and CEDEFOP (mainly of a qualita- 
tive nature and focused on particular issues). There has also been some data collection 
in relation to, or as part of, European cooperation programmes ( Leonardo and Socrates 
and their predecessors). 

Through EUROSTAT, a large number of statistics on European education systems has 
been gathered over the last decades. The “basic” data gathering process in the Member 
States for the European Union, the OECD and UNESCO is undertaken on the basis of 
the same questionnaire to Member States (the so-called UOE-questionnaire ), in order to 
avoid that Member States need to provide the same data several times. Each of these 
bodies processes this data in its own way and for its own purposes. Eurostat also under- 
takes additional surveys that contain data relevant to education and training, such as the 
Labour Force Surveys, the Family Budget and the Eurobarometer surveys. Many of those 
data - where they relate to education and training - are synthesised in the Eurydice pub- 
lications (see below). 

Because of this systematic data gathering process, basic data on initial vocational edu- 
cation has been available for quite some time already, but it has suffered from a number 
of drawbacks. First, the differences between Member States’ vocational education and 
training systems are often hidden in comparative statistics. Secondly, part or whole of 
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vocational education and training may take place after compulsory school age, which 
affects often the availability of data. The same applies to vocational training that is fully or 
predominantly organised outside the traditional school system (e.g. apprentice system). 
Given such methodological and reliability problems, few indicators on initial VET had 
been proposed in the past (since they would have led to false interpretation). As the next 
sections will show, however, there are some recent attempts to address these complexi- 
ties and to construct some reliable indicators. 

As regards continuing vocational education and training , international comparisons are 
of a much more recent nature. Even at national level, the data is often incomplete or even 
lacking. Over the last few years, the Force and Leonardo programmes have been sup- 
porting a number of studies and surveys in order to develop a better basis for interna- 
tional comparisons (see Section 3.3.3 below). 

The next sections describe a number of key trends in relation to the development of com- 
parable VET indicators at EU level. 



3.3.2 Eurydice work on Key data on education in the European Union 

Eurydice is a key education information and analysis network of the European Commis- 
sion and the EU Member States. It consists of a European unit in Brussels, and national 
units (as well as some regional ones) in the EU Member States. Over the last decades, 
the network has developed a track record in terms of analysing differences and similari- 
ties in the EU education systems, and in providing education policy makers with up-to- 
date and reliable information. The Eurydice network is currently sponsored via the Soc- 
rates Programme. 

Over the last few years, increasing attention is being paid to the production of compara- 
ble information about European education systems, both of a descriptive and quantitative 
nature. The most visible output of the trend has been the report “Key data on education in 
the European Union/’, of which the second version (the most recent one when this study 
was finalised) was published in 96 (it is labelled ‘95’, referring to the most recent data 
included) [Eurydice European Unit, 1996], This document contains a wealth of data on 
the current state of the European education systems. 

The report differs from the OECD report Education at a glance in a number of respects: 

■ The information is (currently) limited to EU countries. 

■ Only about 20-25% of the items covered are common with Education at a glance ; in 
fact Eurydice has sought to complement the INES data and has used special 
EUROSTAT surveys as well as specific information gathered via its own network. 

■ The focus of the ‘Key data’ is less on “indicators”, but rather on overall comparisons. 
The quantitative information is displayed in the form of a graph. 

■ Each graph is briefly discussed and the main outcomes are highlighted. 

■ There is a thematic dossier included with very specific and detailed information (the 
1 996 edition concerns teachers). 
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Only a few of the Graphs included relate to vocational education. These are listed below: 



E3 Percentages of pupils in General and Vocational Upper Secondary Education 

E6 Changes in the numbers of girls per 100 boys in upper secondary vocational 
education 

E8 Number of girls per 100 boys in upper secondary vocational education 

El 1 Annual number of class hours in upper secondary vocational education 

E23 Number of girls (per 100 boys) obtaining vocational upper secondary school 
leaving certificates 

J1 Contractual working time and annual number of teaching hours [also for vocational 
education at upper secondary level] 



There are also a few performance or quality related figures: 

A8 Unemployment and level of education (no distinction between general and 
vocational education) 

D1 1 Average duration of primary education (in comparison with theoretical education 
E20 Percentages of 20-year-olds holding a certificate of upper secondary education 
F10 Percentages of 30-year-olds holding a higher education qualification 
G1 Educational expenditure as a percentage of GDP 
G3 Expenditure per pupil as a percentage of per capita GDP 

12 Movement in pupil/teacher ratios by level of education in the public sector 



Interestingly, most of these figures take the form of an indicator (unlike most of the other 
items in the report). 

The third, 1997 version of the report will also cover a number of Central European and 
Baltic countries. The special theme will concern the financing of education, involving all 
the different actors (central, regional and local authorities, educational institutions, pupils, 
households, firms; not-for-profit organisations, etc.) as well as the forms of financing 
(grants, donations, school fees, scholarships, loans, salaries etc.). The specific quantita- 
tive information would be based on the use of (1) the financial part of the renewed UOE 
questionnaire, and (2) EUROSTAT’S Community surveys (Household Panel and Family 
Expenditure surveys). These surveys aim to record over a given period (week, month or 
year) the total consumer expenditure of households, including the expenditure on edu- 
cation. It is as yet unclear to what extent the specificity of vocational education 
(particularly the financing of apprenticeship schemes) would be adequately taken into 
account. 

From 1997 onwards, Eurydice will adopt a new policy. Every other year (i.e. in 1999, 
2001, 2003,...) there would be a ‘core’ publication with the most standard statistics and 
indicators (i.e. comparable with the current versions, bit without the thematic dossiers). 
This document would remain relatively stable as regards the content, which has been 
defined in close cooperation with the Member States. The main developments envisaged 
for the ‘core’ publication are the introduction of data from Central and Eastern European 
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countries, and information about ‘special’ education (for handicapped pupils). In the 
other years (1998, 2000, 2002, ...) the ‘Key data’ report would focus on a special theme. 



3.3.3 European VET-related indicators 

In Section 3.3.1 we mentioned already that the production of internationally comparable 
indicators and statistics regarding vocational education and training has been more 
problematic than for general education. As regards initial vocational education, Section 
3.3.2 showed that some basic European statistics are included in the 'Key data’ docu- 
ment. But comparable data on some aspects which are highly relevant and specific to 
vocational education - such as the transition between school and work (a topic which 
also the OECD is working on), the involvement of the social partners and the relevance of 
the vocational qualifications for the labour market - are hardly available at EU level (at 
national level, the situation is often somewhat better). 

The situation is even more difficult in the field of continuing vocational education and 
training. A first attempt to develop some indicators resulted in the 1994 Force publication 
“Tableau de bord de la formation professionnelle continue" (Force was a European co- 
operation programme on CET, which is now integrated in the Leonardo da Vinci pro- 
gramme) [Commission Europeenne, 1994], Despite numerous methodological difficul- 
ties, a first set of European indicators was produced: 



1. Percentage of employees receiving training 

2. Average duration of training per employee 

3. Average duration of training per trained employee 

4. Average cost of training per employee 

5. Average cost of training per employee trained 

6. Relative cost of training in relation to total salary mass 



The report also included a first proposal for a common framework of indicators on CET. 
These are listed below (instead of ‘CET’ we used the term ‘training’ throughout). 



1. % of employees benefiting from legal training leave arrangements 

2. % of collective employment agreements including a clause on access and 
participation in training 

3. % of participants in on-the-job and off-the-job training 

4. Average duration of on-the-job and off-the-job training 

5. Breakdown of training activities by target group, size of enterprise, type of training, 
level, ... 

6. % of enterprises employing training staff 

7. % of enterprises with training activities /with a training plan 

8. Volume of training (training hours) 

9. Average duration of training by employee, enterprise size, type of training, sector, ... 
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10. % of training provided internally/externally 

11. % of training within or outside working hours 

12. % of training leading to a recognised qualification 

13. Overall spending on training (and in relation to GDP). 

1 4. Average cost of training by position, type of training, sector 

15. Training as a % of the overall salary mass (and differences, e.g. by enterprise 
size and sector) 



At that stage, precise indicators and methods still needed to be worked out. 

Since the launch of the Leonardo da Vinci programme, EUROSTAT and DG XXII intensi- 
fied their efforts to expand and refine the indicators available in the field of VET. In this 
regard, a major publication has been planned for late 1997 - early 1998: "Key Data on 
Vocational Training in the European Union/’ [Eurostat, to be published]. When our study 
was finalised (Summer 1997) the publication was not yet finalised, but the main structure 
had been fixed. Five Chapters will be included: 

A: Demographic trends, educational attainment and the labour market 

B: Initial Vocational Education and Training Programmes 

C: Continuing Vocational Training in Enterprises 

D: Self-employed: participation in training in the past four weeks 

E: The European Community Programmes and Initiatives 

Like in Eurydice’s ‘Key data’ document, the information of this new publication is based 
on a number of surveys and data collection processes from Eurostat in cooperation with 
the EU Member States. 

Given the focus of our work, it seems relevant to list here the full list of indicators planned 
(please note that the list and the definitions were provisional at the time of writing): 



Demographic trends, educational attainment and the labour market 

A1 Trend in the median age of the EU population 
A2 Changes in the age structure of the population of working age 
A3a Educational attainment: comparison by age groups 

A3b Changes in educational attainment at upper-secondary level: comparison of two 
age groups 

A4a Employment rates by educational attainment and sex, 30 to 59 year olds 
A4b Employment rates by educational attainment, 30 to 59 year olds 
A5 Unemployment rates by educational attainment and sex, 30 to 59 year olds 
A6 Young people between 16 and 18 years old in education or not in education 
A7 Unemployment rates by educational attainment and sex, 20 to 29 year olds 
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BEST COPY AVAILABLE 



A8 Unemployment rates: comparison between those with only basic education and 
those completing additional vocational education and training (VET), 20 to 29 year 
olds 

A9 Vulnerability to unemployment: comparison between those with only basic 

compulsory education and those completing post-compulsory VET, 20 to 29 year 
olds 

Initial Vocational Education and Training (IVET) Programmes 

B1 Young people aged 15-19 in education by type of education. 

B2 IVET programmes (16 graphs) 

B3 Participation rates in (IVET) among the population 

B3a Participation rates among the 15-19 year olds population (% of population in IVET) 
B3b Participation rates among the 20-24 year olds population (% of population in IVET) 
B4 Distribution by age group (% of participants in IVET) 

B5 Distribution by gender (% of participants in IVET) 

B6 Where IVET takes place (% of participants in IVET) 

B7 Duration of IVET programmes (% of participants) 

B8 Distribution by ISCED level (% of participants in IVET) 

B9a Access to programmes of a higher level (% of participants in IVET) 

B9b Type of access to programmes at a higher level (% of all participants in IVET) 

B1 0a Certification (in % of participants in IVET) 

BlOb Funding (in % of participants in IVET) 

Continuing Vocational Training (CVT) in Enterprises 

Cl Percentage of enterprises, employing more than 10 people, providing CVT 
C2 Percentage of enterprises providing CVT by enterprise size 
C3 Percentage of enterprises which provided CVT by sector 
C4 Percentage of enterprises providing CVT with training plans /budgets 

C5 Percentage of enterprises providing CVT with training plans /budgets by size 

C6 Percentage of enterprises which provided CVT plans/budgets by sector 
C7 Forms of CVT offered by enterprises providing CVT 
C8 Subjects of CVT 

C9 Percentage distribution of training hours by type of provider 

Cl 0 Costs of CVT courses as a percentage of total labour costs of all enterprises 

ClOa Average CVT cost per participant 

Cl Ob Average CVT cost per participant in “Purchasing Power Parity” 

Cl 1 Participant rates in CVT courses by enterprise size (all enterprises & enterprises 
providing CVT) 
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Cl 2 Average time spent on CVT courses by participants by enterprise size 

Cl 3 Participation rates in CVT courses by sector (all enterprises & enterprises 
providing CVT) 

Cl 4 Time spent by participants on CVT courses by sector 
Cl 5 Percentage of employees participating in CVT courses 
Cl 6 Average time spent on CVT Courses 

Cl 7 Percentage of participants in CVT courses by occupation in all enterprises 
Cl 8 Time spent on CVT courses by occupation for all enterprises 
Cl 9 Percentage of participants (male and female) in CVT courses in all enterprises 
C20 Average duration spent on CVT Courses (male and female) 

Self-employed: participation in training in the past four weeks 

D1 Self-employed as a proportion of all persons in employment 

D2 Distribution of self-employed and employees by sector 

D3 Occupations of self-employed and employees 

D4 Training in the past four weeks - Participation of 30-59 year-olds 

D5 Training in the past four weeks - Participation by educational attainment 

(30-59 year olds) 

D6 Training in the past four weeks - Participation by occupation (30-59 year olds) 

D7 Training in the past four weeks - Participation by age, (30-59 year olds) 

D8 Training in the past four weeks - Participation by gender (30-59 year olds) 

The European Community Programmes and Initiatives 

El European Social Fund (ESF) funds for training 1995-1999 (ECU per inhabitant). 
E2 Leonardo da Vinci Programme and its predecessors - Yearly budget 1987-94 
E3 Leonardo da Vinci committed Funds 1995 and 1 996 



In most cases, the data will relate to 1993-1995. Some concern 12, other 15 EU coun- 
tries. In general, the information will be presented under the form of a graph and/or a ta- 
ble. Please note that the specific indicators on CVT are approximate, since the surveys 
on which they are based are not optimal. 

An inspection of the list shows that most of these indicators are purely ’descriptive’ or 
‘policy’ indicators that say little about effectiveness or quality. However, there are several 
performance indicators, particularly in Chapter C (C1-C6; C10-C16; C20), and some in 
Chapters A and B. Only a limited set of these indicators could really be considered as 
quality indicators, because they say something about the quality of VET provision. This 
concerns in particular the indicators from Chapter A that relate (un)employment to edu- 
cational attainment. Indicators B9a, B9b and BIO could also be considered - from a cer- 
tain perspective - as quality indicators. 
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3.3.4 Other activities at EC level 



Throughout the European Commission, many other DGs are actively involved in the pub- 
lication of international comparisons about a wide range of topics (where it concerns de- 
tailed statistics, this is mostly done in cooperation with EUROSTAT, although other inter- 
national data sources (e.g. OECD) are also used). Often, such comparisons relate to, or 
bear some relevance to education and training. 

It would lead us to far to discuss all these developments, but let us give some examples. 
For instance, there is information available about the perception of youth and families 
about schools. DG V also produces a lot of information about social affairs and training 
for unemployed people. DG XIII has published many European indicators on research 
and development. And DG I is funding a Phare project for the training of Central and 
Eastern European administrators to complete the UOE questionnaires. 



3.4 VET Indicators of the European Training Foundation 

The mandate of the European Training Foundation (ETF) is to promote cooperation be- 
tween the European Union and the ‘Partner’ countries - Central and Eastern Europe, the 
states which previously constituted the Soviet Union, and Mongolia - in vocational train- 
ing, including the coordination of assistance by the Union, its Member States and third 
parties in the sector. More specifically, the European Council has requested the ETF, 
which became operational in 1995, to assist in the definition of training needs and priori- 
ties in cooperation with the relevant authorities. This should include the provision of tar- 
geted information on current initiatives and future needs. 

In order to fulfil this mission, the ETF has launched a number of activities. One of these is 
the establishment of a series of reliable VET indicators on VET, initially in the countries of 
Central and Eastern Europe. This so-called ‘ Key Indicators' project is carried out with the 
assistance of a network of National Observatories that have been established in each of 
the Partner countries. The main aims of this project are: 

■ to provide evidence on the characteristics, problems and challenges of the VET sys- 
tems in the partner countries, and 

■ to measure the VET system’s performance, e.g. in terms of its responsiveness to the 
needs of the labour market or of its capacity to endow people with vocational qualifica- 
tions; 

■ to show - through the regular updating of the key indicators - the trends of VET sys- 
tems and outcomes of implemented policies in the partner states. 

The main strategy adopted was first to establish a limited list of ‘key indicators’ that pro- 
vide basic information on the VET and labour markets in the partner states. The definition 
of the indicators was based on the analysis of the work undertaken by Eurostat, the 
OECD-INES project, and the Employment Observatory on Central and Eastern Europe 
(of DG Vof the European Commission). During the development of the project, work be- 
ing undertaken by other international organisations - i.e. UNESCO and the World Bank - 
was also considered. 

The first set of indicators identified by the ETF was meant to describe the place of VET 
within the socio-economic system of the partner states and measure its effectiveness. 
The list focuses on mainstream vocational education and training - on VET within the 
frame of the regular education and training system - and on labour market trends. This 
basic list of key indicators is as follows: 
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A. Access and Participation in VET 

1. Participation in [general/vocational/higher] education of 14 to 19 year olds. 

2. Participation in VET of 14 to 19 year olds by type of vocational/technical school 

3. Relative weight of vocational education and training at the upper secondary level 

4. Labour market, education and training situation of 16 to 25 year olds 

5. Qualification levels of the population of 25-29 year olds 

B. Outcomes of VET 

6. Drop-outs from Vocational Education and Training 

7. Percentage of graduates of VET to population at theoretical age of graduation 

8. Employment status of those having completed VET courses 

C. Expenditure on VET 

9. Total expenditure on VET 

10. Public expenditure on VET 

1 1 . Public funds for VET by level of government 

D. Employment trends 

12. Employment trends by economic branch 

13. Unemployment overall and by region 

14. Unemployment rates by qualification level of the population 25-59 year olds 



Again, most of these indicators seem to fall under our classification of ‘descriptive’ indi- 
cators, with no direct link to quality issues. In certain contexts, indicators 5. (qualification 
levels of the population), 6. (drop-outs), 8. (employment status) and 13./14. 
(unemployment rates) considered as ‘performance indicators’ or even ‘quality indicators’ 
at macro-economic level. 

At the moment of writing this report, the project was still focusing on these ‘basic indica- 
tors’. Some initial definitions and terminology had been adjusted, and a first publication 
on the indicators available was planned for the end of 1997 - early 1998. This will be a 
very important achievement, given the lack of such comparable figures so far, as well as 
the sometimes very difficult conditions under which some of the data had to be collected. 

In a second phase, this first list would be enhanced and enriched so as to cover specific 
topic areas of interest and to provide a better understanding of the functioning and prob- 
lems of the VET systems. At the stage of finalising this report, it was not yet clear, how- 
ever, to what extent and in what form the efforts would continue beyond 1998, when the 
current funding mechanisms of the Observatory Network will cease to exist. Full exten- 
sion to the former Soviet Union area also seemed problematic. Although it is not yet clear 
to what extent new and more specific indicators would be added in the future, it seems 
useful to mention the areas that were originally envisaged: 
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1. Participation in continuing vocational training 

2. Involvement of enterprises in VET 

3. Further details on expenditure 

4. Labour market trends 

5. Correspondence between qualifications and occupations 

6. Post-secondary VET 

7. Transition from school to work 

8. Participation in VET by field of specialisation 



It is clear that several of these possible areas (particularly the correspondence between 
qualifications and occupations, and the transition from school to work) could lead to 
‘quality indicators’, since they say something about the effectiveness and relevance of 
VET. 

A political issue is also how this ETF initiative would eventually converge and comple- 
ment other international developments in the field of education and training indicators, in 
particular those of Eurostat, Eurydice, OECD and Unesco. 



3.5 Other international developments 

3.5.1 The Third International Maths and Science Study (TIMSS) 

The Third International Maths and Science Study (TIMSS), undertaken by the Interna- 
tional Association for the Evaluation of Educational Achievement (IEA), is one of the few 
large-scale International comparisons which has sought to measure and compare edu- 
cational outcomes - in this case student achievement in mathematics and science. When 
this report was being written, the first results of this third survey had been published 
(including as part of OECD indicator set), although not all results had been processed 
yet. TIMSS is considered to be the largest-ever international education research, involv- 
ing almost 550 000 pupils in three classes of 13 000 schools in 41 countries (including 
most Western and Central European countries). Amongst the central outputs are two in- 
dicators showing the relative performance of 13-year old pupils in Mathematics and Sci- 
ence respectively. The indicator figures show the relative performance compared to the 
average (which is given a default value of 500). The real political indicator — which has 
received wide attention in the press - was the ranking of countries resulting from the in- 
dicator values. 



Although these indicators can hardly be regarded as ‘VET indicators’, they are certainly 
quality indicators. An interesting observation is that the initial ‘hostility’ of teachers and 
governments (fearing embarrassment) for such international comparisons has been de- 
clining over the last decade. More public authorities seem to be interested in knowing 
how their national school system compares with others - even if the results indicate pub- 
licly that performance is low. Indeed, more countries took part in TIMSS than ever before, 
and there is growing evidence that politicians take these indicators very seriously - even 
if they are unflattering for their country. 



An interesting outcome of the existence of the TIMSS indicators, of which the reliability is 
no longer seriously disputed, is that it allows to examine which factors seem to play a role 
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in achieving high quality achievements. Without going into detail here, it is interesting to 
note that the value for the TIMSS indicators does not seem to be directly correlated with 
factors like the number of teaching hours, the state spending per pupil or even class size. 
Such findings illustrate that well-constructed indicators can be useful in exposing “myths’ 
about quality factors - and hence contribute to the policy debate about how quality 
should be improved. 



3.5.2 The International Adult Literacy Survey (IALS) project 

In 1994 seven countries (including Germany, Switzerland, the Netherlands, Sweden and 
Poland) launched the first “ International Adult Literacy Survey" (IALS). Literacy was de- 
fined in terms of the degree of understanding and the use of written material. Levels of 
literacy were defined and related to a range of factors. The processing of the survey re- 
sults has already produced a number of indicators, some of which have been or will be 
included in the OECD indicator set. It is clear that the indicators which relate the adult 
literacy to the educational level, can be considered as rough macro-economic quality 
indicator of the education system. 

This survey is in many ways complementary to TIMSS, both in its content and the popu- 
lation examined. On the other hand, the number of countries participating is still small, 
and the underlying concepts and classifications need still further development. 



3.5.3 World Bank 

The World Bank, through its many support activities relating to education and training in 
developing countries, has also developed interest in the gathering and analysis of edu- 
cation and training statistics as a basis for policy analysis and formulation. On a Euro- 
pean level, it is useful to mention the ‘Social Challenges of Transition” (SCT) project, 
which examined trends in Central and Eastern Europe. The SCT project was initiated in 
order to document empirically the evolving effects of transition on individuals and fami- 
lies. The project has been monitoring indicators of health, earnings, education, labour 
markets, pensions, social assistance and poverty. 

In the field of education, the SCT project has led to the report "Trends in Education Ac- 
cess and Financing during the Transition in Central and Eastern Europe" [Laporte & Rin- 
gold, 1997], This cross-country study examines empirical trends in access to, and fi- 
nancing of, education in nine Central and East European countries (Albania, Bulgaria, 
the Czech Republic, the FYR Macedonia, Hungary, Poland, Romania, Slovakia and 
Slovenia). The education indicators used within the report are the following (similar indi- 
cators have been grouped): 



1 . Enrolment rates (pre-school - basic - secondary - higher) 

2. Secondary enrolments by type (general - technical - vocational) 

3. Enrolments in university programmes by type (engineering - social sciences) 

4. Private school enrolments 

5. Ratio of students to teaching staff (pre-school - basic - secondary - higher) 

6. Education expenditure (in relation to GDP - versus total public expenditure) 

7. Education expenditures by category (capital - non-personnel - personnel) 
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8. Per pupil spending relative to GDP per capita (pre-school - basic - secondary - 
university) 

9. Reduction in real public spending for education 

10. Expenditures on student transfers 

1 1 . Female share of the labour force 

12. Employment changes (increase/decrease per main sector) 

13. Employment in the education sector 

14. Ratio of average wage in the education sector to average wage in the economy 

15. Relation between unemployment and education level 



This World Bank study, together with the work of the European Training Foundation on 
Key Indicators, is of great help in understanding the development of the education sys- 
tem in the countries concerned. Again, however, only the indicator on the relation be- 
tween unemployment and education level could be considered - from a particular quality 
perspective - as a quality indicator, because it gives an indication (but not more than 
that!) about how well an education system is adapted to the needs of the economy. 



3.5.4 Other developments 

There are of course many international organisations and associations that, to a different 
degree and for various purposes, collect and analyse data for policy analysis and per- 
formance measurement. In the field of education, the most noteworthy organisation to be 
mentioned is of course, UNESCO. UNESCO collects educational statistics on a world- 
wide scale, and publishes these regularly. The practical and methodological problems 
faced in this exercise are, of course, even more complex than at OECD or EU level. The 
focus is more on ‘statistics’ than on indicators. Not surprisingly, it is not possible to 
achieve the same level of detail, reliability and comparability as at EU level. UNESCO is 
also the ‘owner’ of the international ISCED' classification of education levels (7 in total). 
It should be noted that UNESCO has put forward recently a proposal to modify the classi- 
fication, in order to obtain a better match with the current education system. Once 
adopted internationally, this should improve the comparability of data. It should also fa- 
cilitate the reliability of VET data and their comparison. All of this is likely to be a stimulus 
for the broader acceptance of education and training indicators. 

The International Labour Organisation (ILO) is also very active in the field of information 
collection and analysis, with a focus, of course, on employment related issues. Some of 
the work bears some direct relationship with education and training issues, such as the 
ILO work on the working conditions of the teaching profession. Also, the ILO is interested 
in examining such things as the quality of employment, the right to training, and the rec- 
ognition of in-company training. It is too early to judge, however, which types of interna- 
tionally accepted indicators might emerge from these developments. 

Let it finally be mentioned that many national governments undertake occasional inter- 
national comparisons between a number of countries, sometimes including quantifiable 
comparisons. This is often motivated by a desire to verify certain hypotheses in relation to 
perceived weaknesses or quality problems. Analysing this area is, however, beyond the 
scope of this study. 
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3.6 



Conclusion 



The previous paragraphs have illustrated how, over the last years, the availability of in- 
ternational indicators in the area of education and training has significantly improved. In 
particular at European level, there is now a vast amount of information available. How- 
ever, most of these focus on the regular, mainstream education system. Vocational edu- 
cation and training is mostly dealt with as part of, and within the frame of, the 
(government funded) education system. Continuing (vocational) education and training is 
only marginally covered, although a number of interesting developments are taking place 
(cf. the forthcoming OECD and EUROSTAT publications). But here there is still a long 
way to go. 

It has also become clear that most international VET or education indicators available are 
of a descriptive nature. As this Chapter showed, however, a number of performance and 
quality indicators have been proposed, but they suffer from a number of drawbacks. In 
general, these indicators show only overall relationships between particular variables, 
and hence their possible contribution to quality improvement activities is limited. Moreo- 
ver, comparability is limited and the accuracy of data often leaves much to be desired. 
Indeed, information on the outcomes of education and training is particularly difficult to 
obtain. Last but not least, the development of more specific quality indicators may be- 
come problematic in terms of acceptance by the international education and training 
community. Increased controversy can be expected about more sensitive quality indica- 
tors, since views on quality itself may differ, and certain quality indicators tend to favour 
particular types of systems. 

This being said, the TIMSS-experience has clearly demonstrated that international re- 
search on student achievement can result in very useful, and generally accepted evi- 
dence. But the scale and volume of resources involved give an indication of the chal- 
lenge which lays ahead - in particular when we realise that math and science compe- 
tence can be more easily measured than general competences such as problem solving 
skills, synthesis skills, situational intelligence. Similar challenges will need to be ad- 
dressed when other types of outcomes need to be measured and compared. 

There are also many technical and practical problems to be overcome. As long as VET 
systems differ substantially, the value of any international indicators will differ. Moreover, 
the way data are aggregated nationally has a major impact on the type of indicators that 
could eventually be constructed, as well as their usefulness. The planned revision of the 
ISCED system may be helpful in this regard, but it will not solve all compatibility prob- 
lems. Many countries of Central and Eastern Europe, moreover, lack an adequate struc- 
ture to track data on such topics as educational outcomes, curriculum content or teaching 
practice. 

A final remark on this topic. Over the last few years, much energy and resources have 
been spent on the development of international indicators. It cannot be denied that their 
publication has already had an impact, not only in the mass media, but also at the level of 
policy makers. Nevertheless, the ‘communication’ purpose seems still to predominate in 
most cases. Nobody seems to be able to provide an idea of the cost-effectiveness of all 
the efforts that have been undertaken. It is unclear how and to what extent the indicators 
are being used for policy development, and whether this leads to quality and perform- 
ance improvement. 
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Chapter 4: Quality indicators of VET providers 



4.1 Introduction 

Common characteristics of the international indicators considered in the previous chapter 
were that they were constructed at the level of education and training systems. Their re- 
lation to quality improvement must therefore be considered at the macro-economic level. 
Some of these indicators could also be useful at the level of an individual VET provider 
(vocational school, training centre, CET institute, ...) and used for different purposes. 
However, at the level of an organisation it is in general more appropriate to develop indi- 
cators that reflect particular quality characteristics or quality goals of the institute 
(although part of these could be common with other institutes). 

Such types of indicators are the focus of this chapter. It should first of all be stated that the 
adoption of quality indicators for VET providers is still in its early development stages. 
Certainly, there is not yet some kind of ‘movement’ in this regard - less even an initial 
consensus. What can be witnessed, however, is the increased recognition of the impor- 
tance of quality approaches and the use of factual evidence in the management of VET 
institutions. Such trends are likely to favour the gradual use of quality indicators. In some 
countries or regions, this trend is already supported by the public authorities, who have 
sometimes ‘suggested’ or imposed the use of particular performance or quality indica- 
tors. In the EU this is as yet, however, more the exception than the rule. 

In view of this study, we have examined a number of cases on the use of quality indica- 
tors with VET providers of different European countries; on some of these we report on 
briefly hereafter. It may be useful to report first on how these cases relate to the 
“Message-Purpose model" which was presented in the first Chapter. On the ‘Message’ 
dimension we encountered all types of indicators; in the sections which follow we will 
concentrate on quality and performance indicators. As regards the ‘purpose’ dimension, 
the whole range was equally covered. Here it is possible to distinguish two ‘clusters’: 

■ process related quality indicators have in general a communication or assessment 
purpose; 

■ goal related quality indicators have most often an assessment or normative purpose. 

In many cases these clusters coincide with a second possible classification: 

■ institution specific quality indicators ; 

■ government or customer driven quality indicators for institutions. 

A third useful classification is: 

■ objectively measurable quality indicators that are derived from verifiable measure- 
ments; 

■ subjective quality indicators that are based on subjective assessment. 

All of this will become more obvious from the examples given. 

Let it also be mentioned that, currently, the use of indicators seems to be increasing in 
private business and other organisations. In the first chapter we mentioned already the 
use recent use of the so-called Balanced Score Card. This is a measurement system, 
including performance indicators, which considers strategic objectives and operational 
performance of an organisation across four perspectives: financial, customers, internal 
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processes, and growth and learning. Adaptations of these methods have already been 
made in various sectors across the world, including in the public sector. The value of 
these - and many similar methods - as a strategic quality improvement methodology is 
increasingly recognised in the business world. Although we did not encounter during our 
research a VET organisation that has already applied such concepts, we thank the un- 
derlying principles and methods could also usefully be applied in a training environment. 
This requires, however, a capacity of institutions to determine their key processes and 
their strategic development goals. 

The examples presented in the next sections do not pretend to be the ‘best ones' or to 
give a ‘complete picture’. Our main aim was to illustrate the wide variety of approaches 
that are currently being used (for that purpose, some higher education cases were also 
included). For each of the examples, a short description is given, and relevant quality 
indicators are listed. We recognise that this brief presentation often does not do justice to 
the overall quality system of the institutions, since the indicators are only a small part of it. 



4.2 Examples of institutional-level VET quality indicators 

4.2.1 A quality system jointly developed by VET institutes in the 
Nijmegen area 

In the early 1990s, a group of VET institutes in the Nijmegen (NL) area developed a 
common approach to quality management and quality assurance. The output of their joint 
efforts was a model for a detailed quality manual and a proposal for a “total quality ap- 
proach” which covered virtually all key areas of VET institutes: Education and Training 
programmes, Input, throughput and output of students, Learning process, Organisation, 
Policy, Staff, Finance, Infrastructure, and Services [Nijmeegs Instituut voor Beroepson- 
derwijs, 1993], For each of these areas key components were defined (58 in total), and to 
each of these was allocated: (1) a quality target or objective, (2) a measurement method, 
and (3) a model for a possible action plan. 

In most of the 58 areas, the combination of a quality target and a measurement led to the 
definition of a quality indicator which could serve as the basis for comparison and further 
action. Byway of example, we list the indicators for two areas: (1) Learning process, and 
(2) Staff. 



Learning process 

1 . % of teachers who consider the learning process to be conform with the educational 
concept of the institute 

2. (a) % of tutoring staff (for placements/practice) who consider that the programme is 
relevant for the chosen employment sector ; (b) % of former students, employed in 
the sector concerned, who consider the programme to be relevant for their 
professional needs 

3. % of teachers, staff and students who consider the programme goals to be clear and 
concretely formulated 

4. % of teachers, staff and students who are satisfied with the integration of theory and 
practice 

5. % of teachers and students who consider that the programme content corresponds 
with the stated learning goals 
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6. % of teachers and students who consider the structure of the programme (content, 
time, sequence) to be clear 

7. % of students who consider that the planned study load corresponds with reality 

8. % of students who are satisfied with the student guidance and advice facilities 

9. % of students of particular target groups who consider that the programme is 
sufficiently adapted to their particular needs 

10. % of students of particular target groups who consider that the programme is 
sufficiently flexible in terms of timing, content and format 

11. % of students who are satisfied with the quality of the programme (several aspects: 
content, didactics, support material, exercises, tests, ...) 

1 2. % of students and practice/placement staff who are satisfied with the practical 
component of the programme (preparation, effect, organisation, clarity of learning 
goals, guidance, evaluation, ...) 

Staff 

1. Absenteeism (compared with national average) 

2. Proportion between male and female full-time equivalent (norm = 50/50) 

3. Correspondence between qualitative/quantitative needs for staff and the actual 
situation 

4. % of teaching staff time spent on teaching 

5. number of departments and programmes in which teaching staff teaches 
(goal: minimum) 

6. % of staff who consider that their work load corresponds with their appointment 

7. frequency and intensity of staff contacts / employment in the employment sector 

8. level of insight in the training needs of staff 

9. opinion on support for incoming and outgoing staff 

1 0. frequency of staff appraisals 



It is obvious from this (incomplete) list that the indicator set contains measures of very 
different kind, both objective and subjective ones. 



4.2.2 VET indicators and quality criteria of the Region Emilia-Romagna 
in Italy 

The Department for Vocational Training of the regional government of the Emilia- 
Romagna region in Italy has been pursuing an active quality policy in relation to the 
training it funds. Since the early 1990s they have defined quality criteria for training or- 
ganisations and for training programmes. The authorities have recently even taken a 
bold step by requiring that, by the end of 1998, any training organisation will need to 
have been 'accredited' on he basis of strict quality criteria, if it wishes to remain eligible 
for public support for vocational training programmes. 

Amongst the quality criteria for VET programmes, used for the ex ante evaluation and 
approval of programmes, the region has defined a priority set of 4 criteria. These criteria 



include one or more indicators (summarised below), which are expressed as a value on 
a scale of 1 to 5, corresponding with the extent to which a certain quality characteristic 
has been met[Regione Emilia-Romagna, 1992]: 

Criterion 3: Relevance with regard to labour market needs and 
development projects 

1 . Relevance in relation to sectoral, geographical or developmental needs 

2. Importance of the problems to be addressed 

3. Involvement of socio-economic actors 

Criterion 4: Relevance with regard to the needs of potential users 

4. Adequacy and potential take-up with regard to the real needs of potential users 

Criterion 8: Accuracy and adequacy of the overall training programme 

5. Consistency and details of the training content in relation to the objectives 

6. Adequacy of the training approach (didactics, duration, student assessment, ...) 

7. Consistency between training content and the organisation put into place (trainers, 
materials, ...) 

Criterion 11: Existence of a documented professional profile (only for initial 
training) 

8. Comprehensiveness of the description of the professional situation targeted 

9. Detailed description of the competence profile 

10. Validation of the competence profile by employers' representatives 



By their very nature, these indicators are subjective and can only be attributed by quali- 
fied training experts during the ex ante evaluation of programme proposals. A detailed 
manual has been developed, explaining the value of the scale items (1 - 5) for each of 
the different indicators. The measures are an example of “normative” quality indicators, 
since they are used for selection and compare a (complex) variable with a given stan- 
dard. Please note that these indicators, unlike most other ones that are given in this Sec- 
tion, relate to programmes rather than to an institution, and only concern input elements 
(they do not measure trainee satisfaction, for instance). 



4.2.3 A Portuguese pilot case on Quality in a vocational school 

EPED (Vocational School for Development) is a relatively recent vocational institute that 
emerged following a European PETRA cooperation project. It was initially focused on the 
area of environment management but has since widened its scope [Ambrosio & Andrade, 
1995]. 

The school pursues an active quality policy that has led to the establishment of a Quality 
Observatory that has started the development of quality indicator in areas like: 
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1 . Adequacy of human resources 

2. Quality of material resources 

3. Management and organisation of the training 

4. Evaluation 

5. Degree of trainee satisfaction 

6. Trainee adaptability to the employment post 

7. Possibilities of acquiring local apprenticeships 

8. Possibilities of acquiring employment 



4.2.4 PROZA- a comprehensive self-assessment tool for education 
institutes 

During 1996-97, a group of Flemish higher (mainly vocational) education institutes de- 
veloped a comprehensive self-assessment tool, “PROZA”, that could be useful for self- 
assessment, analysis and quality improvement of institutions [Van den Berghe e.a. 
(forthcoming)]. The overall framework with 9 main areas was inspired by the EFQM- 
model for quality and business excellence (Leadership, Policy and Strategy, etc.). The 
system furthermore contains almost 90 quality criteria (grouped under the 9 headings) 
covering virtually every key aspect of the input, organisation and outputs of higher 
(vocational) education institutions. For every criterion, five quality stages were defined 
(like in the Emilia-Romagna case). In addition, for every stage a checklist of four or five 
questions was established, in order to facilitate self-assessment. In total, the system in- 
cludes some 2000 questions, which can be used with a variety of methodologies. 

One use of the PROZA-system is the construction of quality indicators for each of the cri- 
teria, or for groups of these. The quality indicator corresponds with the highest quality 
stage achieved for an individual criterion. For brevity, we list the current (Summer 1997) 
quality criteria of the 5th and 7th areas. 



5. Core processes 

5.1 Process control - general 

5.2 Programme and curriculum development 

5.3 Development of courses and other programme components 

5.4 Planning and interaction of course schedules 

5.5 Programme implementation 

5.6 Teaching and direct teacher-student interaction 

5.7 Tutoring and guided learning 

5.8 Professional contacts and placements 

5.9 Didactic support material 

5.10 Student guidance and support 

5. 1 1 Assessment of student achievement 

5.12 Continuing education 
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7. Student satisfaction 

7.1 Educational infrastructure 

7.2 Attitude of staff 

7.3 Social services 

7.4 Information provision 

7.5 Participation possibilities 

7.6 Complaint handling 

7.7 Learning load 

7.8 Consistency of the programme 

7.9 Didactic approach 

7.10 Guidance 



The PROZA-system is already being adapted for other contexts, e.g. for the vocational 
training of handicapped people. 



4.2.5 The Danish Strategic Quality Plan for VET 

In 1995 the Department of Vocational Education and Training of the Danish Ministry of 
Education presented a systematic quality strategy plan. This was based on the experi- 
ence gained with a wide range of quality initiatives undertaken by VET institutes over the 
previous years. The plan had several components, and paid considerable attention to 
relevant statistics, as well as to the development of supplementary quality criteria and 
indicators [Nielsen, S., 1995], 

In the Strategy Plan, indicators were announced concerning the internal quality work of 
the vocational institutes in the following areas: 



Strategy Development 

Management instruments 

1 . Strategic management, including school profiling through the supply of courses and 
services 

2. Adaptation strategies to changing goals and needs 

3. Local educational plans and curricular work 

4. Drawing up of budgets 

Educational instruments 

5. Students’ right to be consulted / participation 

6. School culture and environment 

External contacts 

7. Cooperation with local education and training committees 



ERIC 



42 



8. Collaboration with other schools and colleges: locally, regionally and nationally 

9. International activities 

Resource parameters 

Allocation 

1 0. Planning of supply of courses and services 

1 1 . Economic management and cash-flow control 

12. Staff recruiting and policy 

13. Equipment and physical facilities (incl. library) 

14. Registration of students’ throughput, incl. completion rates 

Operational aspects 

15. Guidance, introduction 

16. Special educational assistance 

17. Safety/working environment 

18. Organisation of examinations 

19. LOP-registration and “out-reaching” activities (incl. educational offers for adults) 

Innovative and development activities 

20. Organisation of learning, incl. differentiation, adaptation to the needs of students, 
integrated teaching through interdisciplinary learning approaches, etc. 

21 . Human resources policies, incl. continuing training of teachers 

22. Innovation of education and development work 



For all these areas quality criteria are being developed as well as indicators and meth- 
ods to measure good or bad quality. 



4.2.6 The quality scoreboard of the “Centrum Industrleel Beleid”, 
KULeuven 

The Centrum Industrieel Beleid of the Katholieke Universiteit Leuven is a small university 
department which is - in addition to research - involved in providing advances level 
continuing education and training in the field of industrial management. The Centre was 
one of the first to start with a quality system approach, which led to the award of an ISO 
9001 certificate in 1994. In 1993 the Centre started also with the development of an indi- 
cator system [De Bruyn & Gelders, 1994], The initial purpose was to define measurable 
quality objectives. For that purpose, quality norms were defined for critical processes, 
and indicators were defined in order to measure to what extent the norm is attained. The 
most difficult part appeared to be the definition of the norm. In some cases, it was rela- 
tively easy to define a standard, in general in the form of a minimum or maximum value. 
In other cases, the average of the last three years was taken as the initial norm. The indi- 
cator set has been adapted and refined over the years. 
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In the field of advanced continuing education and training programmes, the current qual- 
ity indicators of the Centre are: 

Enrolment 

1 . Number of requests for enrolment by September 1 

2. Number of accepted students 

3. Number of students enrolled by November 1 

4. Number of Masters degree students enrolled 

5. % of successful students for the July and September examinations 

6. Number of degrees awarded/number of students enrolled 

7. % op optional courses which students take within the Centre 

Student satisfaction (based on student surveys) 

8. Average score on ‘satisfaction’ of the programme 

9. Average score on ‘importance’ of the programme 

10. Number of courses which over 10% of students consider as inappropriate 
(norm = 0) 

1 1 . Average score on ‘satisfaction’ of the thesis/term paper 

1 2. Average score on ‘importance’ of the thesis/term paper 

13. % of seminars with average overall score below 6.5 (max =10) (norm = 1 0%) 

Theses/Term papers 

14. number of submitted theses/term papers in July and September 

15. number of companies which have not been involved 

16. number of students not finalising their thesis/term paper 

17. number of theses/term papers undertaken by a single student 



As can be seen, certain indicators are absolute numbers. The full set of indicators of the 
Centre, which indicate the performance with regard to the quality objectives, form to- 
gether a scoreboard which is used for the management review of the quality system (a 
requirement of ISO 9001). It may be useful to mention that the interpretation and expla- 
nation of the values of the indicators (and the trends) may be more important than the 
values themselves. Some indicators (e.g. 15, 17) reflect quality characteristics which are 
very specific to the programme taught (in other circumstances, these would be purely 
descriptive indicators). 



4.2.7 Murdoch University (Western Australia) 




This university has developed a strategic plan and goals in the areas of teaching and 
learning, research, community service, management, and the university profile [Murdoch 
University, 1997], Key performance indicators have been defined to measure achieve- 
ments in these areas. In the area of teaching, the goal is “to provide a challenging 
teaching and learning partnership which attracts able students and produces graduates 
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who are valued and sought after by employers". The associated quality and performance 
indicators are: 



1 . Student progress rate (passed subject load expressed as a proportion of certified 
subject load) 

2. Programme completion rate (% of students completing programmes) 

3. Cohort retention rate (proportion of students who continue from year 1 to year 2) 

4. Graduates’ perceived quality of teaching (average values on six areas resulting 
from a national survey) 

5. Graduate status (% of graduates in employment or in full-time study at a specified 
date in the first year following completion of the course 

6. Student evaluation of teaching (% of students of students satisfied) 

7. Employers’ perception of quality (% considering graduates to be ‘good’ or 
‘excellent’) 

8. Recurrent expenditures per successful equivalent Full-time Student Unit 



4.3 Categories of indicators 

4.3.1 Typology and attributes 

In Chapter 2 we presented our general model for the classification of indicators, the so- 
called "Message-Purpose” model. As the previous examples have illustrated, it is possi- 
ble to classify most indicators with this model. Let us illustrate this with the following table, 
giving an example for each cross-section of the “Message-Purpose" model: 



Examples of classification of indicators in the “Message-Purpose” Model 


Purpose 

Message 


Analysis 


Communication 


Normative 


Quality 


Extent of relevance of 
the training pro- 
gramme 


Trainee satisfaction 
with regard to train- 
ing 


Unemployment rate 
following VET 
courses 


Performance 


% of planned courses 
delayed or cancelled 


Average time be- 
tween initial idea 
and implementation 
of training 


Average cost of a 
training course 


Policy/ 

Management 


Trends in percent- 
ages of accepted ap- 
plicants 


% of employees fol- 
lowing training 


Average educational 
profile of trainees 


Description 


Male/female distribu- 
tion of staff & trainees 


Average number of 
topics per course 


Minimum number of 
trainees per course 
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